Java Tip: Consuming SharePoint web services with a Java client

Using CAML for CRUD methods on SharePoint Copy and Lists services

I used to think that a Camel was something to smoke or ride in the desert, but that was before I opened my mouth at work one day and said, "Sure I can automatically save these documents up to SharePoint." SharePoint has been around for a long time, so I assumed that there must be a Java API or possibly some exposed web services that I could consume. Well, as it turns out I did get the job done, and as with many things it ended better than it started. But there were quite a few hurdles along the way, which I'd like to help other JavaWorld readers avoid.

In this Java tip I'll show you how to perform basic CRUD operations on a SharePoint document folder from a Java client. The demonstration will focus on some of the more popular methods that any SharePoint web service consumer is likely to invoke, which are found in Microsoft's Copy and Lists services. For the CRUD operations we'll use CAML (Collaborative Application Markup Language), an XML-based language utilized in many of the methods exposed by Copy and Lists. You'll learn how to construct valid CAML structures that are passed as method parameters or assigned to object properties, which are in turn passed as parameters to these services.

Hopefully, this tip will convince you that there are CAMLs that you can use to get business documents from point A to point B, without relying on four legs and a hump.

Demonstration code

My demonstration code is very simple: I've used no open source code other than for logging and my implementation has no dependencies on Java EE technology, so you can run the source code directly from Eclipse in a standard Java application.

Background

I work in a group that performs information management for large volumes of data that eventually is housed in various data-marts (service, export, report, and so on). Consumers, both internal and external to the company, consume the data in order to make business and personal decisions. Various kinds of monitoring take place in this environment, including automated audits and reports that are run against the data housed in data-marts. Audits ensure that data is in a consistent state, both across marts and within the mart where it resides. Audit reports are emailed to various people and then manually saved on SharePoint.

Because the engines used for monitoring have a pluggable output writer concept, it was natural to consider setting up a writer for SharePoint. We already were writing to the database, SMTP servers, and a file system, so this seemed like a logical next step and a way to avoid the manual process.

The trick, of course, was making it all work.

Getting started: Communicating with SharePoint

The sample application for this article demonstrates how to communicate with SharePoint from a Java client. I wrote the application using Eclipse 3.6.2 and Java 1.6.0_32. Figure 1 shows the two main packages contained within the sample application.

Figure 1. Package diagram (click to enlarge)

The first package, com.jw.sharepoint.examples, contains all the custom code for the solution. It uses the code contained in the com.microsoft.sharepoint.webservices package, which was code-generated.

Before diving into how the custom code is structured I'll explain how to generate the Microsoft package. First, recall that we'll be using two web services to invoke service calls: Copy and Lists. You can access these services on the SharePoint site that you are trying to communicate with in the following locations:

  • https://server/site/_vti_bin/Lists.asmx
  • https://server/site/_vti_bin/Copy.asmx

Generating the web services package

To generate the code for the web services package we'll use wsimport, which is located in the bin directory of your Java installation, assuming that you're using Java 1.6 or higher. If your SharePoint site is running over HTTPS you might have a problem running wsimport when pointing it directly to your server via the above URLs, in which case you would receive an error like this one:

[ERROR] sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPath
BuilderException: unable to find valid certification path to requested target

The problem in this case is that your cacerts file doesn't have the certificate from the site. An easy way to get around this is to use your browser to download the WSDL files locally. For the example listed below I have done just that and saved the WSDLs in c:\temp\. Listing 1 and Listing 2 show the code snippets I used to generate the source code, along with the output. You can ignore the warning for each service.

Listing 1. Copy service code generation

C:\temp>"%JAVA_HOME%\bin\wsimport" -d . -p com.microsoft.schemas.sharepoint.soap -keep -extension -Xnocompile Copy.wsdl
parsing WSDL...

[WARNING] SOAP port "CopySoap12": uses a non-standard SOAP 1.2 binding.
  line 229 of file:/C:/temp/Copy.wsdl

generating code...

Listing 2. Lists service code generation

C:\temp>"%JAVA_HOME%\bin\wsimport" -d . -p com.microsoft.schemas.sharepoint.soap -keep -extension -Xnocompile list.wsdl
parsing WSDL...

[WARNING] SOAP port "ListsSoap12": uses a non-standard SOAP 1.2 binding.
  line 1511 of file:/C:/temp/list.wsdl

generating code...

Once you've generated the code it's ready to be incorporated into the solution and used. You can remove the –Xnocompile option from the wsimport command. This option would cause the class files to be generated along with the source, but for this exercise we'll just copy the generated source files into the solution and let Eclipse compile them as if we had authored the source code.

A note about security

In order to successfully execute the SharePoint services I had to deviate from my normal method of web service consumption, which most always involves the use of Axis2. I quickly found that Axis2 has issues with the NTML authorization. It's possible to overcome these errors by using JCIFS in conjunction with Axis2 (see Resources) but that seemed like overkill for something relatively easy. With the approach I'm demonstrating there are no security hurdles to overcome. If your SharePoint site is using HTTPS you will need to ensure that the cacerts file is updated with the site’s certificate (see Resources for details).

Because the examples are meant to be executed as console applications in Eclipse, I pass the following VM argument in the run configuration:

-Djavax.net.ssl.trustStore=path to your updated cacerts file

Custom code

The custom code for this solution is located in the com.jw.sharepoint.examples package in the article source code. It contains a custom class for each of the SharePoint functions we'll be testing:

  1. SharePointUploadDocumentExample demonstrates how to upload a document to SharePoint.
  2. SharePointDeleteListItemExample demonstrates how to delete a document from SharePoint using CAML to query a list and delete a list item.
  3. SharePointListExample demonstrates how to query a folder on SharePoint using CAML and then interpret the results.

Note that I won't explicitly discuss the final class, SharePointListExample. The SharePointDeleteListItemExample class contains functionality for querying, so SharePointListExample is presented for you to study on your own.

About the custom classes

As shown in Figure 2, each of the custom classes follows the same pattern and extends the SharePointBaseExample class, which provides basic SharePoint functionality, as well as utility functions for dealing with CAML and XML. The custom classes also use specific properties files that they load via an initialize() function that is called in main. The properties files have all the properties needed for communicating with SharePoint and any other data that is required at runtime for the class in question.

Figure 2. Class diagram for the custom code (click to enlarge)

Properties files

Each of the properties files located in the demonstration code's Configuration directory has the name of the class it supports with a .properties extension. Most of the properties contained in these files should be self-explanatory. Table 1 briefly describes the additional properties contained in SharePointDeleteListItemExample.properties.

Table 1. Additional properties of the SharePointDeleteListItemExample class

PropertyDescriptionExample value
username The username for authentication to the SharePoint site. This should be fully domain qualified if running on Linux or using a different ID than used for Windows authentication. Domain\bigbird
password The password to the SharePoint site sesame
wsdl URL to the Lists.asmx WSDL https://abc.xyz.com/project/epms9615/_vti_bin/Lists.asmx?wsdl
Endpoint URL to the Lists.asmx https://abc.xyz.com/project/epms9615/_vti_bin/Lists.asmx
Folder The name of the base folder to use. Prod Support Folder
copy.wsdl URL to the Copy.asmx WSDL https://abc.xyz.com/team/eds/_vti_bin/Copy.asmx?wsdl
copy.endpoint URL to the Copy.asmx https://abc.xyz.com/team/eds/_vti_bin/Copy.asmx
copy.location The location to place files for upload https://abc.xyz.com/project/epms9615/Prod%20Support%20Folder/
Daily%20Monitoring%20Status/AuditDeleteTesting/
copy.sourceFile The local file to use for uploading Configuration/SharePointDeleteListItemExample.properties
delete.FileRef.base The base URL to the SharePoint site, used in delete file requests. https://abc.xyz.com/

Additional configuration files

Some additional configuration files are located in the Configuration directory. These are simple XML snippets written in CAML. We'll use these files, described in Table 2, throughout the solution.

Table 2. Additional configuration files

CAML fileDescription
Query.xml A CAML file containing the query that we'll use to list files from the SharePoint server. This file shows an example of a query that uses three fields with two different data types (Text and DateTime), as well as two different operators (Contains and Eq).
QueryOptions.xml A static file that we'll use throughout the examples to tell a SharePoint service that we want it to search all subfolders of the current folder.
Delete.xml A CAML file that we'll use to delete SharePoint files. Strings are substituted at runtime.
DeleteListItemQuery.xml A CAML file that we'll use to perform a query of candidate files available for removal from SharePoint

First demo: Uploading a file to SharePoint

Our first exercise will be uploading a file to SharePoint via the CopySoap web service. For this we'll use some of the classes that we generated in Listing 1 and Listing 2 by executing wsimport on the Copy.asmx.

In the SharePointBaseExample class you'll notice a method named getCopySoap(). We'll use this method to return a generated CopySoap instance, which we'll then use to upload a file by calling the method uploadDocument(CopySoap port, String sourceUrl).

The getCopySoap() method is shown in Listing 3.

Listing 3. getCopySoap()

protected CopySoap getCopySoap() throws Exception {
    logger.info("Creating a CopySoap instance...");
    Copy service = new Copy(new URL(getProperties().getProperty("copy.wsdl")),
        new QName("http://schemas.microsoft.com/sharepoint/soap/", "Copy"));
    CopySoap copySoap = service.getCopySoap();
    BindingProvider bp = (BindingProvider) copySoap;
    bp.getRequestContext().put(BindingProvider.USERNAME_PROPERTY, getProperties().getProperty("username"));
    bp.getRequestContext().put(BindingProvider.PASSWORD_PROPERTY, getProperties().getProperty("password"));
    bp.getRequestContext().put(BindingProvider.ENDPOINT_ADDRESS_PROPERTY, getProperties().getProperty("copy.endpoint"));
    return copySoap;
}

A Copy class is instantiated using a two-argument constructor, which takes the copy service's WSDL location along with the QName instance to use. We get the CopySoap instance we need from the Copy instance. Once this is done we can cast it to a BindingProvider, which performs the protocol binding and contains the associated context objects for request-and-response message processing. From the BindingProvider we can then set the username, password, and endpoint information on the request-context Map.

Listing 4 shows the main method of class SharePointUploadDocumentExample. This method is very simple; it uses getCopySoap() and uploadDocument(CopySoap port, String sourceUrl) to upload a document to SharePoint. The source file to be copied to SharePoint is defined in the SharePointUploadDocumentExample associated properties file, which it passes to the uploadDocument(…) method via the copy.sourceFile property value.

Listing 4. Upload document main method

public static void main(String[] args) {
    logger.debug("main...");        
    try {       
        SharePointUploadDocumentExample example = new SharePointUploadDocumentExample();
        example.initialize();
        CopySoap p = example.getCopySoap();
        example.uploadDocument(p, properties.getProperty("copy.sourceFile"));
    } catch (Exception ex) {
        logger.error("Error caught in main: ",ex);
    }
}

uploadDocument()

Next we'll call the uploadDocument() method. There are a few things to know about this method:

  1. Properties pertaining to the document such as the Title can be set by adding a FieldInformation instance to the FieldInformationCollection. I've demonstrated this in Listing 5, where the title is set to the source-file name.
  2. You can upload a document to multiple locations by adding more than one destination to the DestinationUrlCollection instance. The configured properties file specifies a second location to be used.
  3. When looping through the CopyResultCollection (which is populated after making the service call) status codes and messages are returned for each destination URL that you've supplied. You can then check the result for each destination specified.
  4. If the file is versioned on SharePoint, uploading it with the uploadDocument() method will update the document along with its properties and increase the version number. Thus you can use the uploadDocument() method for updates as well as initial file uploads.

Listing 5. The uploadDocument() method

protected  void uploadDocument(CopySoap port, String sourceUrl)   throws Exception {

    File f = new File(sourceUrl);
    logger.debug("Uploading: " + f.getName());

    String url = getProperties().getProperty("copy.location") + f.getName();
    DestinationUrlCollection destinationUrlCollection = new DestinationUrlCollection();
    destinationUrlCollection.getString().add(url);
    if(getProperties().getProperty("copy.location2") != null){
         url = getProperties().getProperty("copy.location2") + f.getName();
         destinationUrlCollection.getString().add(url);
    }
        

    FieldInformation titleFieldInformation = new FieldInformation();
    titleFieldInformation.setDisplayName("Title");
    titleFieldInformation.setType(FieldType.TEXT);
    titleFieldInformation.setValue(f.getName());

    FieldInformationCollection fields = new FieldInformationCollection();
    fields.getFieldInformation().add(titleFieldInformation);

    CopyResultCollection results = new CopyResultCollection();
    Holder resultHolder = new Holder(results);
    Holder longHolder = new Holder(new Long(-1));
        
    //make the call to upload
    port.copyIntoItems(sourceUrl, destinationUrlCollection, fields, readAll(f), longHolder,resultHolder);
        
    //does not seem to change based on different CopyResults
    logger.debug("Long holder: " + longHolder.value);
        
    //do something meaningful here
    for (CopyResult copyResult : resultHolder.value.getCopyResult()) {              
        logger.debug("Destination: " + copyResult.getDestinationUrl());
        logger.debug("Error Message: " + copyResult.getErrorMessage());
        logger.debug("Error Code: " + copyResult.getErrorCode());
        if(copyResult.getErrorCode() != CopyErrorCode.SUCCESS)
            throw new Exception("Upload failed for: " + copyResult.getDestinationUrl() + " Message: " 
                    + copyResult.getErrorMessage() + " Code: " +   copyResult.getErrorCode() );
    }

}

Results

In the output shown in Listing 6, below, you can see that I have forced an error by trying to upload a document to two locations where the second location does not exist. This error helps me illustrate how to interrogate a CopyResult for each destination.

Listing 6. CopyResult output

2013-09-09 15:16:26,805 [main] DEBUG examples.SharePointBaseExample - Destination: https://abc.xyz.com/project/epms9615/Prod%20Support%20Folder/Daily%20Monitoring%20Status/AuditDeleteTesting/SharePointUploadDocumentExample.properties
2013-09-09 15:16:26,805 [main] DEBUG examples.SharePointBaseExample - Error Message: null
2013-09-09 15:16:26,806 [main] DEBUG examples.SharePointBaseExample - Error Code: SUCCESS
2013-09-09 15:16:26,806 [main] DEBUG examples.SharePointBaseExample - Destination: https://abc.xyz.com/project/epms9615/Prod%20Support%20Folder/Daily%20Monitoring%20Status/AuditDeleteTesting/Folder2/SharePointUploadDocumentExample.properties
2013-09-09 15:16:26,806 [main] DEBUG examples.SharePointBaseExample - Error Message: Object reference not set to an instance of an object.
2013-09-09 15:16:26,806 [main] DEBUG examples.SharePointBaseExample - Error Code: UNKNOWN
2013-09-09 15:16:26,808 [main] ERROR examples.SharePointUploadDocumentExample - Error caught in main: 
java.lang.Exception: Upload failed for: https://abc.xyz.com/project/epms9615/Prod%20Support%20Folder/Daily%20Monitoring%20Status/AuditDeleteTesting/Folder2/SharePointUploadDocumentExample.properties Message: Object reference not set to an instance of an object. Code: UNKNOWN
    at com.jw.sharepoint.examples.SharePointBaseExample.uploadDocument(SharePointBaseExample.java:178)
    at com.jw.sharepoint.examples.SharePointUploadDocumentExample.main(SharePointUploadDocumentExample.java:23)

Second demo: Deleting files on SharePoint using web services

Before I can show you how to delete a file on SharePoint I need to briefly demonstrate how to query SharePoint using the web services. The idea is that you will issue a query and retrieve a list of documents that are stored on SharePoint. Once you have the list you will be able to delete one or more of the documents found by the query operation. The code for this section is found in the SharePointDeleteListItemExample class. The main() method for this class is very similar to SharePointUploadDocumentExample's main(): it uploads a document to SharePoint, sends a query for the uploaded document, and ultimately deletes the document. I have added two additional lines to this class's main() method to achieve the query and delete functionality:

  1. ListsSoap ls = example.getListsSoap();
  2. example.executeQueryAndDelete(ls);

Querying a folder

In the previous example we used CopySoap to upload a document. For this example we'll use ListsSoap to query a folder and remove some of the documents it contains. The getListsSoap() method looks exactly like the getCopySoap() method, with the exception of the class it ultimately returns to the caller, so I won't detail it here.

The QueryOptions instance shown in Listing 7 is configured to have the getListItems() method recurse through all sub-folders on SharePoint for a given query. We achieve this by adding a static XML structure on the QueryOptions instance, by calling the getContent().add(Object o) method.

Listing 7. Query options


    TRUE
     
    TRUE 

The actual query the SharePoint web service will execute is defined by an org.w3c.dom.Node instance and is again a CAML XML block. Listing 8 shows an example of what the CAML query looks like at runtime, querying multiple document properties. In this example we query documents using three properties: Editor, FileRef, and Created_x0020_Date. For the Editor and FileRef properties we use a Contains operator. For the Created_x0020_Date property we use an Eq (equal) operator.

Listing 8. Query


    
    
        
             
                 
                Chandler
            
             
                 
                AuditDeleteTesting
                 
        
         
             
            2013-09-10
                       
              
    

In Listing 9 the static CAML XML structures are populated with data and then set on the Query and QueryOptions instances. Note the ListsSoap.getListItems(…) method parameters as well as the GetListItemsResult returned by the method.

Listing 9. Execute query

public void executeQueryAndDelete(ListsSoap ls) throws Exception {
    Date today = Calendar.getInstance().getTime();
    SimpleDateFormat simpleDateFormat = new SimpleDateFormat("yyyy-MM-dd");
    String formattedDate = simpleDateFormat.format(today);
    String queryFormatted = String.format(deleteListItemQuery,formattedDate);       
        
    GetListItems.QueryOptions msQueryOptions = new GetListItems.QueryOptions();
    GetListItems.Query msQuery = new GetListItems.Query();
    msQuery.getContent().add(createSharePointCAMLNode(queryFormatted));
    msQueryOptions.getContent().add(createSharePointCAMLNode(this.queryOptions));
    GetListItemsResponse.GetListItemsResult result = ls.getListItems(
            properties.getProperty("folder"), "", msQuery, null, "",
            msQueryOptions, "");

    writeResult(result.getContent().get(0), System.out);

    Element element = (Element) result.getContent().get(0);
    NodeList nl = element.getElementsByTagName("z:row");
    for (int i = 0; i < nl.getLength(); i++) {
        Node node = nl.item(i);
        String id = node.getAttributes().getNamedItem("ows_ID").getNodeValue();
        String fileRefRelativePath = node.getAttributes().getNamedItem("ows_FileRef").getNodeValue();
        logger.debug("id: " + id);
        logger.debug("fileRefRelativePath: " + fileRefRelativePath);
        String fileRef = properties.getProperty("delete.FileRef.base") + fileRefRelativePath.split("#")[1];
        logger.debug("fileRef: " + fileRef);
        deleteListItem(ls, properties.getProperty("folder"), id, fileRef);
    }
}

Listing 9 uses two different CAML structures, represented as org.w3c.dom.Node instances via the QueryOptions and Query instances. After those objects have the appropriate CAML structures set the getListItems method is invoked and in turn uses these structures to query the SharePoint site. The results are examined within the loop at the bottom of the function.

In Listing 10, notice that the GetListItemsResult instance contains result content in XML format, which can be cast to an Element instance for processing. In this example, we process all the nodes with the tag name of z:row. These nodes contain the metadata regarding the documents that were found as a result of executing the query defined by the CAML structure in Listing 8.

We use specific metadata from each row to aid in deleting specific documents returned from the query. Specifically, we use the ows_ID and ows_FileRef. Listing 10 is an abbreviated version of the XML returned from the query. It contains the namespace declarations as well as many of the attributes returned for the sake of brevity.

Listing 10. Query result XML

 

Deleting a list item from a folder

For the delete operation we'll use the ListsSoap.updateListItems(…) method shown in Listing 11.

Listing 11. Delete list item

public void deleteListItem(ListsSoap ls, String listName, String listId, String fileRef) throws Exception {
    String deleteFormatted = String.format(delete, listId, fileRef);        
    Updates u = new Updates();
    u.getContent().add(createSharePointCAMLNode(deleteFormatted));
    UpdateListItemsResult ret = ls.updateListItems(listName, u);
    writeResult(ret.getContent().get(0), System.out);
}

The function in Listing 11 also takes a CAML structure, which is shown in Listing 12. To successfully delete a file you must specify the ID which was returned in the query result XML as ows_ID for the file in question as well as the FileRef.

Listing 12. Delete XML structure


    
                1739
                        https://abc.xyz.com/project/epms9615/Prod Support Folder/Daily Monitoring Status/AuditDeleteTesting/SharePointDeleteListItemExample.properties
                  

The response returned isn't as friendly the CopyResult returned, in that it doesn't contain getters that can be invoked for error messages and status codes. But it does contain all the necessary data that you would interrogate in order to determine whether or not the call was successful on the server. A couple of XML structured responses are shown in Listing 13 and Listing 14.

Listing 13. Delete response (success)



    
        0x00000000
    

Listing 14. Delete response (failure)



    
        0x81020030
        Invalid file name

The file name you specified could not be used.  It may be the name of an existing file or directory, or you may not have permission to access the file.
        
    

Notice that when the FileRef filed information is missing (see Listing 14) the response returns an embedded z:row that contains all the information for the file in question based on the ID that we passed in Listing 12. You might wonder in that case why both the ID and FileRef parameters are required in the delete request. I suppose a CAML can only do so much.

In conclusion

In this Java tip I have showed you how to use CAML to execute standard CRUD methods on documents stored on SharePoint, based on my own experience of automating file uploads to SharePoint. I believe this tip should be very useful to Java developers because most companies that I have worked or consulted for utilized SharePoint for storing documents. In addition to using the demonstration classes, I hope that you will find other practical uses for the techniques described in this article. For instance, you could easily incorporate the process that I have demonstrated into build scripts, using it to write and save build results for easy access as well as the results of test automation. Any business process that has manual steps involving SharePoint or does not currently involve SharePoint, but does produce documents that must be saved, versioned, or easily accessible could be considered a candidate for the approach described in this tip.

Joe Chandler is a senior architecture specialist at Cigna in Denver, Colorado. He has more than 20 years of experience in design and development of both Java EE and .NET-based solutions with a strong focus in geospatial solutions due to his tenure at Autodesk. He holds a BS degree in Computer Information Systems from University of Colorado, Boulder, and an MS in Information Science from the University of Denver.

Learn more about this topic

  • Download the source code for this Java tip.
  • Visit the Java tips page for more useful tips from working Java programmers.
  • Programming with CAML (SharePoint Dev Center): Get more tutorials in customizing SharePoint sites and services with CAML.
  • Learn more about Microsoft's Copy and Lists web services for SharePoint.
  • "Reading a SharePoint list with Java" (David's IT blog, February 2010): Get another tutorial on using external business data with SharePoint and a Java client.
  • Several forum questions helped to inform this Java tip, including these about Java GetList Items (SharePoint Dev Center) and uploading to SharePoint (StackOverflow.com).
  • Another StackOverflow.com thread summarizes the experience of getting JCIFS to play with AXIS (May 2009), while a thread on the SharePoint Stack Exchange offers tips for resolving it (July 2011).
  • If your SharePoint site is using HTTPS you will need to ensure that the cacerts file is updated with the site's trusted certificate.
Join the discussion
Be the first to comment on this article. Our Commenting Policies
See more