Package org.frankframework.filesystem

Class AmazonS3FileSystem

java.lang.Object

org.frankframework.filesystem.AbstractFileSystem<S3FileRef>

org.frankframework.filesystem.AmazonS3FileSystem

All Implemented Interfaces:

AutoCloseable, HasPhysicalDestination, IBasicFileSystem<S3FileRef>, ISupportsCustomFileAttributes<S3FileRef>, IWritableFileSystem<S3FileRef>

@DestinationType(FILE_SYSTEM)
public class AmazonS3FileSystem
extends AbstractFileSystem<S3FileRef>
implements IWritableFileSystem<S3FileRef>, ISupportsCustomFileAttributes<S3FileRef>

Field Summary

Fields inherited from interface org.frankframework.filesystem.ISupportsCustomFileAttributes

FILE_ATTRIBUTE_PARAM_PREFIX

Constructor Summary

Constructors

Constructor	Description
AmazonS3FileSystem()

Method Summary

Modifier and Type	Method	Description
void	appendFile(S3FileRef file, InputStream content)
void	close()
void	configure()
S3FileRef	copyFile(S3FileRef s3Object, String destinationFolder, boolean createFolder)	Copies the file to another folder.
S3FileRef	copyFile(S3FileRef s3Object, String destinationFolder, boolean createFolder, Map<String,String> customFileAttributes)	Copies the file to another folder, adding the given custom file attributes to the file in the destination folder.
void	createFile(S3FileRef f, InputStream content)
void	createFile(S3FileRef f, InputStream content, Map<String,String> customFileAttributes)	Creates a file with the given custom file attributes.
void	createFolder(String folder)
software.amazon.awssdk.services.s3.S3Client	createS3Client()
void	deleteFile(S3FileRef f)
boolean	exists(S3FileRef f)
boolean	folderExists(String folder)
Map<String,Object>	getAdditionalFileProperties(S3FileRef f)
String	getCanonicalName(S3FileRef f)
protected software.amazon.awssdk.auth.credentials.AwsCredentialsProvider	getCredentialProvider()
String	getCustomFileAttribute(S3FileRef file, String name)	Gets the value of the given custom file attribute on the given file.
long	getFileSize(S3FileRef f)
protected software.amazon.awssdk.http.apache.ApacheHttpClient.Builder	getHttpClientBuilder()
Date	getModificationTime(S3FileRef f)
String	getName(S3FileRef f)	Get a string representation of an identification of a file.
int	getNumberOfFilesInFolder(String folder)
String	getParentFolder(S3FileRef f)
String	getPhysicalDestinationName()
boolean	isFolder(S3FileRef s3Object)
DirectoryStream<S3FileRef>	list(S3FileRef folder, TypeFilter filter)
S3FileRef	moveFile(S3FileRef f, String destinationFolder, boolean createFolder)	Moves the file to another folder.
S3FileRef	moveFile(S3FileRef f, String destinationFolder, boolean createFolder, Map<String,String> customFileAttributes)	Moves the file to another folder, adding the given custom file attributes to the file in the destination folder.
void	open()
Message	readFile(S3FileRef file, String charset)	If you retrieve an S3Object, you should close this input stream as soon as possible, because the object content is not buffered in memory and streams directly from Amazon S3.
void	removeFolder(String folder, boolean removeNonEmptyFolder)
S3FileRef	renameFile(S3FileRef source, S3FileRef destination)	Renames the file to a new name, possibly in another folder.
S3FileRef	renameFile(S3FileRef source, S3FileRef destination, Map<String,String> customFileAttributes)	Renames the file to a new name, possibly in another folder.
void	setAccessKey(String accessKey)	Access key to access to the AWS resources owned by the account
void	setAuthAlias(String authAlias)	Alias used to obtain AWS credentials
void	setBucketName(String bucketName)	Name of the bucket to access.
void	setChunkedEncodingDisabled(boolean chunkedEncodingDisabled)	Setting this flag will result in disabling chunked encoding for all requests.
void	setClientRegion(String clientRegion)	Name of the AWS region that the client is using.
void	setForceGlobalBucketAccessEnabled(boolean forceGlobalBucketAccessEnabled)	Option to enable or disable the usage of multi-region access point ARNs.
void	setMaxConnections(int maxConnections)	Maximum concurrent connections towards S3
void	setProxyHost(String proxyHost)	Proxy host
void	setProxyPort(Integer proxyPort)	Proxy port
void	setSecretKey(String secretKey)	Secret key to access to the AWS resources owned by the account
void	setServiceEndpoint(String serviceEndpoint)	The S3 service endpoint, either with or without the protocol.
void	setStorageClass(S3StorageClass storageClass)	Set the desired storage class for the S3 object when action is move,copy or write.
S3FileRef	toFile(String filename)	Creates a local S3Object pointer, not representative with what is stored in the S3 Bucket.
S3FileRef	toFile(String folder, String filename)	Creates a reference to a file.

Methods inherited from class org.frankframework.filesystem.AbstractFileSystem

getMaxNumberOfMessagesToList, isOpen, setMaxNumberOfMessagesToList

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.frankframework.filesystem.IBasicFileSystem

getCanonicalNameOrErrorMessage, isOpen, list

Methods inherited from interface org.frankframework.filesystem.ISupportsCustomFileAttributes

getCustomFileAttributes, hasCustomFileAttributes

Constructor Details

AmazonS3FileSystem

public AmazonS3FileSystem()

Method Details

configure

public void configure()
               throws ConfigurationException

Specified by:

configure in interface IBasicFileSystem<S3FileRef>

Throws:

ConfigurationException

open

public void open()
          throws FileSystemException

Specified by:

open in interface IBasicFileSystem<S3FileRef>

Overrides:

open in class AbstractFileSystem<S3FileRef>

Throws:

FileSystemException

getCredentialProvider

protected software.amazon.awssdk.auth.credentials.AwsCredentialsProvider getCredentialProvider()

createS3Client

public software.amazon.awssdk.services.s3.S3Client createS3Client()

close

public void close()
           throws FileSystemException

Specified by:

close in interface AutoCloseable

Specified by:

close in interface IBasicFileSystem<S3FileRef>

Overrides:

close in class AbstractFileSystem<S3FileRef>

Throws:

FileSystemException

toFile

public S3FileRef toFile(@Nullable
 String filename)

Creates a local S3Object pointer, not representative with what is stored in the S3 Bucket. This method may be used to upload a file to S3.

Specified by:

toFile in interface IBasicFileSystem<S3FileRef>

toFile

public S3FileRef toFile(@Nullable
 String folder,
 @Nullable
 String filename)

Description copied from interface: IBasicFileSystem

Creates a reference to a file. If filename is not absolute, it will be created in 'defaultFolder'.

Specified by:

toFile in interface IBasicFileSystem<S3FileRef>

getNumberOfFilesInFolder

public int getNumberOfFilesInFolder(String folder)
                             throws FileSystemException

Specified by:

getNumberOfFilesInFolder in interface IBasicFileSystem<S3FileRef>

Overrides:

getNumberOfFilesInFolder in class AbstractFileSystem<S3FileRef>

Throws:

FileSystemException

list

public DirectoryStream<S3FileRef> list(S3FileRef folder,
 TypeFilter filter)
                                throws FileSystemException

Specified by:

list in interface IBasicFileSystem<S3FileRef>

Throws:

FileSystemException

exists

public boolean exists(S3FileRef f)
               throws FileSystemException

Specified by:

exists in interface IBasicFileSystem<S3FileRef>

Throws:

FileSystemException

isFolder

public boolean isFolder(S3FileRef s3Object)

Specified by:

isFolder in interface IBasicFileSystem<S3FileRef>

createFile

public void createFile(S3FileRef f,
 InputStream content)
                throws FileSystemException,
IOException

Specified by:

createFile in interface IWritableFileSystem<S3FileRef>

Parameters:

f - FileSystem file reference

content - to write or NULL. When NULL existing files should be overwritten, and new files should be created.

Throws:

FileSystemException

IOException

createFile

public void createFile(S3FileRef f,
 InputStream content,
 Map<String,String> customFileAttributes)
                throws FileSystemException,
IOException

Description copied from interface: ISupportsCustomFileAttributes

Creates a file with the given custom file attributes.

Specified by:

createFile in interface ISupportsCustomFileAttributes<S3FileRef>

Parameters:

f -

content -

customFileAttributes -

Throws:

FileSystemException

IOException

appendFile

public void appendFile(S3FileRef file,
 InputStream content)
                throws FileSystemException,
IOException

Specified by:

appendFile in interface IWritableFileSystem<S3FileRef>

Throws:

FileSystemException

IOException

readFile

public Message readFile(S3FileRef file,
 String charset)
                 throws FileSystemException

If you retrieve an S3Object, you should close this input stream as soon as possible, because the object content is not buffered in memory and streams directly from Amazon S3. Failure to close this stream can cause the request pool to become blocked.

Specified by:

readFile in interface IBasicFileSystem<S3FileRef>

Throws:

FileSystemException

deleteFile

public void deleteFile(S3FileRef f)
                throws FileSystemException

Specified by:

deleteFile in interface IBasicFileSystem<S3FileRef>

Throws:

FileSystemException

folderExists

public boolean folderExists(String folder)
                     throws FileSystemException

Specified by:

folderExists in interface IBasicFileSystem<S3FileRef>

Throws:

FileSystemException

createFolder

public void createFolder(String folder)
                  throws FileSystemException

Specified by:

createFolder in interface IBasicFileSystem<S3FileRef>

Throws:

FileSystemException

removeFolder

public void removeFolder(String folder,
 boolean removeNonEmptyFolder)
                  throws FileSystemException

Specified by:

removeFolder in interface IBasicFileSystem<S3FileRef>

Throws:

FileSystemException

renameFile

public S3FileRef renameFile(S3FileRef source,
 S3FileRef destination)
                     throws FileSystemException

Description copied from interface: IWritableFileSystem

Renames the file to a new name, possibly in another folder. Does not need to check for the existence of the source or non-existence of the destination.

Specified by:

renameFile in interface IWritableFileSystem<S3FileRef>

Throws:

FileSystemException

renameFile

public S3FileRef renameFile(S3FileRef source,
 S3FileRef destination,
 Map<String,String> customFileAttributes)
                     throws FileSystemException

Description copied from interface: ISupportsCustomFileAttributes

Renames the file to a new name, possibly in another folder. Does not need to check for the existence of the source or non-existence of the destination.

Specified by:

renameFile in interface ISupportsCustomFileAttributes<S3FileRef>

Throws:

FileSystemException

copyFile

public S3FileRef copyFile(S3FileRef s3Object,
 String destinationFolder,
 boolean createFolder)
                   throws FileSystemException

Description copied from interface: IBasicFileSystem

Copies the file to another folder. Does not need to check for existence of the source or non-existence of the destination. Returns the copied file, or null if no file was copied or there is no reference to the copied file.

Specified by:

copyFile in interface IBasicFileSystem<S3FileRef>

Throws:

FileSystemException

copyFile

public S3FileRef copyFile(S3FileRef s3Object,
 String destinationFolder,
 boolean createFolder,
 Map<String,String> customFileAttributes)
                   throws FileSystemException

Description copied from interface: ISupportsCustomFileAttributes

Copies the file to another folder, adding the given custom file attributes to the file in the destination folder. Does not need to check for existence of the source or non-existence of the destination. Returns the copied file, or null if no file was copied or there is no reference to the copied file.

Specified by:

copyFile in interface ISupportsCustomFileAttributes<S3FileRef>

Throws:

FileSystemException

moveFile

public S3FileRef moveFile(S3FileRef f,
 String destinationFolder,
 boolean createFolder)
                   throws FileSystemException

Description copied from interface: IBasicFileSystem

Moves the file to another folder. Does not need to check for existence of the source or non-existence of the destination. Returns the moved file, or null if no file was moved or there is no reference to the moved file.

Specified by:

moveFile in interface IBasicFileSystem<S3FileRef>

Throws:

FileSystemException

moveFile

public S3FileRef moveFile(S3FileRef f,
 String destinationFolder,
 boolean createFolder,
 Map<String,String> customFileAttributes)
                   throws FileSystemException

Description copied from interface: ISupportsCustomFileAttributes

Moves the file to another folder, adding the given custom file attributes to the file in the destination folder. Does not need to check for existence of the source or non-existence of the destination. Returns the moved file, or null if no file was moved or there is no reference to the moved file.

Specified by:

moveFile in interface ISupportsCustomFileAttributes<S3FileRef>

Throws:

FileSystemException

getAdditionalFileProperties

@Nullable
public Map<String,Object> getAdditionalFileProperties(S3FileRef f)

Specified by:

getAdditionalFileProperties in interface IBasicFileSystem<S3FileRef>

getName

public String getName(S3FileRef f)

Description copied from interface: IBasicFileSystem

Get a string representation of an identification of a file. Must pair up with the implementation of IBasicFileSystem.toFile(String). Can reflect name a file has in its folder, is not expected to be unique over folders.

Specified by:

getName in interface IBasicFileSystem<S3FileRef>

getParentFolder

public String getParentFolder(S3FileRef f)

Specified by:

getParentFolder in interface IBasicFileSystem<S3FileRef>

getCanonicalName

public String getCanonicalName(S3FileRef f)

Specified by:

getCanonicalName in interface IBasicFileSystem<S3FileRef>

getFileSize

public long getFileSize(S3FileRef f)
                 throws FileSystemException

Specified by:

getFileSize in interface IBasicFileSystem<S3FileRef>

Throws:

FileSystemException

getModificationTime

public Date getModificationTime(S3FileRef f)
                         throws FileSystemException

Specified by:

getModificationTime in interface IBasicFileSystem<S3FileRef>

Throws:

FileSystemException

getHttpClientBuilder

protected software.amazon.awssdk.http.apache.ApacheHttpClient.Builder getHttpClientBuilder()

getCustomFileAttribute

@Nullable
public String getCustomFileAttribute(@Nonnull
 S3FileRef file,
 @Nonnull
 String name)

Description copied from interface: ISupportsCustomFileAttributes

Gets the value of the given custom file attribute on the given file. If the attribute does not exist, null is returned.

Specified by:

getCustomFileAttribute in interface ISupportsCustomFileAttributes<S3FileRef>

Parameters:

file - File object for which to get the extended attribute. Must not be null.

name - Name of the extended attribute to get. Must not be null.

Returns:

Value of the extended attribute or null if the attribute does not exist.

getPhysicalDestinationName

public String getPhysicalDestinationName()

Specified by:

getPhysicalDestinationName in interface HasPhysicalDestination

setAccessKey

public void setAccessKey(String accessKey)

Access key to access to the AWS resources owned by the account

setSecretKey

public void setSecretKey(String secretKey)

Secret key to access to the AWS resources owned by the account

setAuthAlias

public void setAuthAlias(String authAlias)

Alias used to obtain AWS credentials

setChunkedEncodingDisabled

public void setChunkedEncodingDisabled(boolean chunkedEncodingDisabled)

Setting this flag will result in disabling chunked encoding for all requests.

Default value

false

setForceGlobalBucketAccessEnabled

public void setForceGlobalBucketAccessEnabled(boolean forceGlobalBucketAccessEnabled)

Option to enable or disable the usage of multi-region access point ARNs.

Default value

false

setServiceEndpoint

public void setServiceEndpoint(String serviceEndpoint)

The S3 service endpoint, either with or without the protocol. (e.g. https://sns.us-west-1.amazonaws.com or sns.us-west-1.amazonaws.com)

setClientRegion

@Mandatory
public void setClientRegion(String clientRegion)

Name of the AWS region that the client is using.

Default value

eu-west-1

setBucketName

public void setBucketName(String bucketName)

Name of the bucket to access. The bucketName can also be specified by prefixing it to the object name, separated from it by "|"

setProxyHost

public void setProxyHost(String proxyHost)

Proxy host

setProxyPort

public void setProxyPort(Integer proxyPort)

Proxy port

setStorageClass

public void setStorageClass(S3StorageClass storageClass)

Set the desired storage class for the S3 object when action is move,copy or write. More info on storage classes can be found on the AWS S3 docs.

Default value

STANDARD

setMaxConnections

public void setMaxConnections(int maxConnections)

Maximum concurrent connections towards S3

Default value

50