nl.nn.adapterframework.filesystem

Class AmazonS3FileSystem

java.lang.Object

nl.nn.adapterframework.filesystem.FileSystemBase<com.amazonaws.services.s3.model.S3Object>

nl.nn.adapterframework.filesystem.AmazonS3FileSystem

All Implemented Interfaces:

HasPhysicalDestination, IBasicFileSystem<com.amazonaws.services.s3.model.S3Object>, IWritableFileSystem<com.amazonaws.services.s3.model.S3Object>

public class AmazonS3FileSystem
extends FileSystemBase<com.amazonaws.services.s3.model.S3Object>
implements IWritableFileSystem<com.amazonaws.services.s3.model.S3Object>

Field Summary

Fields inherited from class nl.nn.adapterframework.filesystem.FileSystemBase

log

Constructor Summary

Constructors

Constructor and Description
AmazonS3FileSystem()

Method Summary

Modifier and Type	Method and Description
OutputStream	appendFile(com.amazonaws.services.s3.model.S3Object f)
void	close()
void	configure()
com.amazonaws.services.s3.model.S3Object	copyFile(com.amazonaws.services.s3.model.S3Object f, String destinationFolder, boolean createFolder, boolean resultantMustBeReturned) Copies the file to a another folder.
OutputStream	createFile(com.amazonaws.services.s3.model.S3Object f)
void	createFolder(String folder)
com.amazonaws.services.s3.AmazonS3	createS3Client()
void	deleteFile(com.amazonaws.services.s3.model.S3Object f)
boolean	exists(com.amazonaws.services.s3.model.S3Object f)
boolean	folderExists(String folder)
Map<String,Object>	getAdditionalFileProperties(com.amazonaws.services.s3.model.S3Object f)
static List<String>	getAvailableRegions()
String	getCanonicalName(com.amazonaws.services.s3.model.S3Object f)
com.amazonaws.ClientConfiguration	getClientConfig()
protected com.amazonaws.auth.AWSCredentialsProvider	getCredentialProvider()
long	getFileSize(com.amazonaws.services.s3.model.S3Object f)
Date	getModificationTime(com.amazonaws.services.s3.model.S3Object f)
String	getName(com.amazonaws.services.s3.model.S3Object f) Get a string representation of an identification of a file.
int	getNumberOfFilesInFolder(String folder)
String	getParentFolder(com.amazonaws.services.s3.model.S3Object f)
String	getPhysicalDestinationName()
DirectoryStream<com.amazonaws.services.s3.model.S3Object>	listFiles(String folder) Lists all files in 'folder' or in the 'root' of the filesystem (when folder is null).
com.amazonaws.services.s3.model.S3Object	moveFile(com.amazonaws.services.s3.model.S3Object f, String destinationFolder, boolean createFolder, boolean resultantMustBeReturned) Moves the file to a another folder.
void	open()
Message	readFile(com.amazonaws.services.s3.model.S3Object file, String charset) If you retrieve an S3Object, you should close this input stream as soon as possible, because the object contents aren't buffered in memory and stream directly from Amazon S3.
void	removeFolder(String folder, boolean removeNonEmptyFolder)
com.amazonaws.services.s3.model.S3Object	renameFile(com.amazonaws.services.s3.model.S3Object source, com.amazonaws.services.s3.model.S3Object destination) Renames the file to a new name, possibly in a another folder.
void	setAccessKey(String accessKey) Access key to access to the AWS resources owned by the account
void	setAuthAlias(String authAlias) Alias used to obtain AWS credentials
void	setBucketName(String bucketName) Name of the bucket to access.
void	setChunkedEncodingDisabled(boolean chunkedEncodingDisabled) Setting this flag will result in disabling chunked encoding for all requests.
void	setClientRegion(String clientRegion) Name of the region that the client will be created from
void	setForceGlobalBucketAccessEnabled(boolean forceGlobalBucketAccessEnabled) Set whether the client should be configured with global bucket access enabled.
void	setMaxConnections(int maxConnections) Maximum concurrent connections towards S3
void	setProxyHost(String proxyHost) Proxy host
void	setProxyPort(Integer proxyPort) Proxy port
void	setSecretKey(String secretKey) Secret key to access to the AWS resources owned by the account
void	setServiceEndpoint(String serviceEndpoint) The S3 service endpoint, either with or without the protocol.
void	setStorageClass(com.amazonaws.services.s3.model.StorageClass storageClass) Set the desired storage class for the S3 object when action is move,copy or write.
com.amazonaws.services.s3.model.S3Object	toFile(String filename) Creates a local S3Object pointer, not representative with what is stored in the S3 Bucket.
com.amazonaws.services.s3.model.S3Object	toFile(String folder, String filename) Creates a reference to a file.

Methods inherited from class nl.nn.adapterframework.filesystem.FileSystemBase

getMaxNumberOfMessagesToList, isOpen, setMaxNumberOfMessagesToList

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface nl.nn.adapterframework.filesystem.IBasicFileSystem

isOpen

Methods inherited from interface nl.nn.adapterframework.core.HasPhysicalDestination

getDomain

Constructor Detail

AmazonS3FileSystem

public AmazonS3FileSystem()

Method Detail

configure

public void configure()
               throws ConfigurationException

Specified by:

configure in interface IBasicFileSystem<com.amazonaws.services.s3.model.S3Object>

Throws:

ConfigurationException

open

public void open()
          throws FileSystemException

Specified by:

open in interface IBasicFileSystem<com.amazonaws.services.s3.model.S3Object>

Overrides:

open in class FileSystemBase<com.amazonaws.services.s3.model.S3Object>

Throws:

FileSystemException

getCredentialProvider

protected com.amazonaws.auth.AWSCredentialsProvider getCredentialProvider()

createS3Client

public com.amazonaws.services.s3.AmazonS3 createS3Client()

close

public void close()
           throws FileSystemException

Specified by:

close in interface IBasicFileSystem<com.amazonaws.services.s3.model.S3Object>

Overrides:

close in class FileSystemBase<com.amazonaws.services.s3.model.S3Object>

Throws:

FileSystemException

toFile

public com.amazonaws.services.s3.model.S3Object toFile(String filename)
                                                throws FileSystemException

Creates a local S3Object pointer, not representative with what is stored in the S3 Bucket. This method may be used to upload a file to S3.

Specified by:

toFile in interface IBasicFileSystem<com.amazonaws.services.s3.model.S3Object>

Throws:

FileSystemException

toFile

public com.amazonaws.services.s3.model.S3Object toFile(String folder,
                                                       String filename)
                                                throws FileSystemException

Description copied from interface: IBasicFileSystem

Creates a reference to a file. If filename is not absolute, it will be created in 'defaultFolder'.

Specified by:

toFile in interface IBasicFileSystem<com.amazonaws.services.s3.model.S3Object>

Throws:

FileSystemException

getNumberOfFilesInFolder

public int getNumberOfFilesInFolder(String folder)
                             throws FileSystemException

Specified by:

getNumberOfFilesInFolder in interface IBasicFileSystem<com.amazonaws.services.s3.model.S3Object>

Overrides:

getNumberOfFilesInFolder in class FileSystemBase<com.amazonaws.services.s3.model.S3Object>

Throws:

FileSystemException

listFiles

public DirectoryStream<com.amazonaws.services.s3.model.S3Object> listFiles(String folder)
                                                                    throws FileSystemException

Description copied from interface: IBasicFileSystem

Lists all files in 'folder' or in the 'root' of the filesystem (when folder is null). Should list only 'files', no folders.

Specified by:

listFiles in interface IBasicFileSystem<com.amazonaws.services.s3.model.S3Object>

Throws:

FileSystemException

exists

public boolean exists(com.amazonaws.services.s3.model.S3Object f)
               throws FileSystemException

Specified by:

exists in interface IBasicFileSystem<com.amazonaws.services.s3.model.S3Object>

Throws:

FileSystemException

createFile

public OutputStream createFile(com.amazonaws.services.s3.model.S3Object f)
                        throws FileSystemException,
                               IOException

Specified by:

createFile in interface IWritableFileSystem<com.amazonaws.services.s3.model.S3Object>

Throws:

FileSystemException

IOException

appendFile

public OutputStream appendFile(com.amazonaws.services.s3.model.S3Object f)
                        throws FileSystemException,
                               IOException

Specified by:

appendFile in interface IWritableFileSystem<com.amazonaws.services.s3.model.S3Object>

Throws:

FileSystemException

IOException

readFile

public Message readFile(com.amazonaws.services.s3.model.S3Object file,
                        String charset)
                 throws FileSystemException,
                        IOException

If you retrieve an S3Object, you should close this input stream as soon as possible, because the object contents aren't buffered in memory and stream directly from Amazon S3. Further, failure to close this stream can cause the request pool to become blocked.

Specified by:

readFile in interface IBasicFileSystem<com.amazonaws.services.s3.model.S3Object>

Throws:

FileSystemException

IOException

deleteFile

public void deleteFile(com.amazonaws.services.s3.model.S3Object f)
                throws FileSystemException

Specified by:

deleteFile in interface IBasicFileSystem<com.amazonaws.services.s3.model.S3Object>

Throws:

FileSystemException

folderExists

public boolean folderExists(String folder)
                     throws FileSystemException

Specified by:

folderExists in interface IBasicFileSystem<com.amazonaws.services.s3.model.S3Object>

Throws:

FileSystemException

createFolder

public void createFolder(String folder)
                  throws FileSystemException

Specified by:

createFolder in interface IBasicFileSystem<com.amazonaws.services.s3.model.S3Object>

Throws:

FileSystemException

removeFolder

public void removeFolder(String folder,
                         boolean removeNonEmptyFolder)
                  throws FileSystemException

Specified by:

removeFolder in interface IBasicFileSystem<com.amazonaws.services.s3.model.S3Object>

Throws:

FileSystemException

renameFile

public com.amazonaws.services.s3.model.S3Object renameFile(com.amazonaws.services.s3.model.S3Object source,
                                                           com.amazonaws.services.s3.model.S3Object destination)
                                                    throws FileSystemException

Description copied from interface: IWritableFileSystem

Renames the file to a new name, possibly in a another folder. Does not need to check for existence of the source or non-existence of the destination.

Specified by:

renameFile in interface IWritableFileSystem<com.amazonaws.services.s3.model.S3Object>

Throws:

FileSystemException

copyFile

public com.amazonaws.services.s3.model.S3Object copyFile(com.amazonaws.services.s3.model.S3Object f,
                                                         String destinationFolder,
                                                         boolean createFolder,
                                                         boolean resultantMustBeReturned)
                                                  throws FileSystemException

Description copied from interface: IBasicFileSystem

Copies the file to a another folder. Does not need to check for existence of the source or non-existence of the destination. Returns the copied file, or null if no file was copied or there is no reference to the copied file. If the reference to the copied file is unknown after the copy, then:
if resultantMustBeReturned is set, then an Exception must be thrown, preferably before the file is copied;
if resultantMustBeReturned is not set, then a null result returned might also mean the file was copied successfully, but with unknown destination;

Specified by:

copyFile in interface IBasicFileSystem<com.amazonaws.services.s3.model.S3Object>

resultantMustBeReturned - TODO

Throws:

FileSystemException

moveFile

public com.amazonaws.services.s3.model.S3Object moveFile(com.amazonaws.services.s3.model.S3Object f,
                                                         String destinationFolder,
                                                         boolean createFolder,
                                                         boolean resultantMustBeReturned)
                                                  throws FileSystemException

Description copied from interface: IBasicFileSystem

Moves the file to a another folder. Does not need to check for existence of the source or non-existence of the destination. Returns the moved file, or null if no file was moved or there is no reference to the moved file.
If the reference to the moved file is unknown after the move, then:
if resultantMustBeReturned is set, then an Exception must be thrown, preferably before the file is moved;
if resultantMustBeReturned is not set, then a null result returned might also mean the file was moved successfully, but with unknown destination;

Specified by:

moveFile in interface IBasicFileSystem<com.amazonaws.services.s3.model.S3Object>

resultantMustBeReturned - TODO

Throws:

FileSystemException

getAdditionalFileProperties

public Map<String,Object> getAdditionalFileProperties(com.amazonaws.services.s3.model.S3Object f)

Specified by:

getAdditionalFileProperties in interface IBasicFileSystem<com.amazonaws.services.s3.model.S3Object>

getName

public String getName(com.amazonaws.services.s3.model.S3Object f)

Description copied from interface: IBasicFileSystem

Get a string representation of an identification of a file. Must pair up with the implementation of IBasicFileSystem.toFile(String). Can reflect name a file has in its folder, is not expected to be unique over folders.

Specified by:

getName in interface IBasicFileSystem<com.amazonaws.services.s3.model.S3Object>

getParentFolder

public String getParentFolder(com.amazonaws.services.s3.model.S3Object f)
                       throws FileSystemException

Specified by:

getParentFolder in interface IBasicFileSystem<com.amazonaws.services.s3.model.S3Object>

Throws:

FileSystemException

getCanonicalName

public String getCanonicalName(com.amazonaws.services.s3.model.S3Object f)
                        throws FileSystemException

Specified by:

getCanonicalName in interface IBasicFileSystem<com.amazonaws.services.s3.model.S3Object>

Throws:

FileSystemException

getFileSize

public long getFileSize(com.amazonaws.services.s3.model.S3Object f)
                 throws FileSystemException

Specified by:

getFileSize in interface IBasicFileSystem<com.amazonaws.services.s3.model.S3Object>

Throws:

FileSystemException

getModificationTime

public Date getModificationTime(com.amazonaws.services.s3.model.S3Object f)
                         throws FileSystemException

Specified by:

getModificationTime in interface IBasicFileSystem<com.amazonaws.services.s3.model.S3Object>

Throws:

FileSystemException

getClientConfig

public com.amazonaws.ClientConfiguration getClientConfig()

getPhysicalDestinationName

public String getPhysicalDestinationName()

Specified by:

getPhysicalDestinationName in interface HasPhysicalDestination

getAvailableRegions

public static List<String> getAvailableRegions()

setAccessKey

public void setAccessKey(String accessKey)

Access key to access to the AWS resources owned by the account

setSecretKey

public void setSecretKey(String secretKey)

Secret key to access to the AWS resources owned by the account

setAuthAlias

public void setAuthAlias(String authAlias)

Alias used to obtain AWS credentials

setChunkedEncodingDisabled

public void setChunkedEncodingDisabled(boolean chunkedEncodingDisabled)

Setting this flag will result in disabling chunked encoding for all requests.

Default value

false

setForceGlobalBucketAccessEnabled

public void setForceGlobalBucketAccessEnabled(boolean forceGlobalBucketAccessEnabled)

Set whether the client should be configured with global bucket access enabled.

Default value

false

setServiceEndpoint

public void setServiceEndpoint(String serviceEndpoint)

The S3 service endpoint, either with or without the protocol. (e.g. https://sns.us-west-1.amazonaws.com or sns.us-west-1.amazonaws.com)

setClientRegion

@Mandatory
public void setClientRegion(String clientRegion)

Name of the region that the client will be created from

Default value

eu-west-1

setBucketName

public void setBucketName(String bucketName)

Name of the bucket to access. The bucketName can also be specified by prefixing it to the object name, separated from it by

setProxyHost

public void setProxyHost(String proxyHost)

Proxy host

setProxyPort

public void setProxyPort(Integer proxyPort)

Proxy port

setStorageClass

public void setStorageClass(com.amazonaws.services.s3.model.StorageClass storageClass)

Set the desired storage class for the S3 object when action is move,copy or write. More info on storage classes can be found on the AWS S3 docs: https://aws.amazon.com/s3/storage-classes/

Default value

STANDARD

setMaxConnections

public void setMaxConnections(int maxConnections)

Maximum concurrent connections towards S3