File Storage

As the file storage with S3 API we use MinIO.

To work with this file storage via S3 API we have a special utility - StorageManager.

Injecting Storage Manager
Upload file to storage
Download file from storage
Delete file from storage
Get list of files
Complete example

Injecting Storage Manager

Storage Manager can be injected into your task class:

import eu.ibagroup.easyrpa.engine.service.StorageManager;

@Inject
private StorageManager storageManager;

Upload file to storage

To upload the file using File you can use method with the following signature:

public void uploadFile(String bucket, String path, File file)

Where:

bucket - is the target bucket name.
path - is the target file path which includes folders structure, file name and file extension, but without specifying the bucket name. E.g.: "folder1/subfolder/my_file.txt"
file - the file to upload

To upload the file using InputStream you can use method with the following signature:

public void uploadFile(String bucket, String path, InputStream input)

Where:

bucket - is the target bucket name.
path - is the target file path which includes folders structure, file name and file extension, but without specifying the bucket name. E.g.: "folder1/subfolder/my_file.txt"
input - the input stream

The following example demonstrates how to upload file to S3 and get its URL:

String fileName = UUID.randomUUID().toString() + ".txt";
String s3FilePath = "upload_file_test/" + fileName;
byte[] fileContent = "Text file content example".getBytes();

storageManager.uploadFile("data", s3FilePath, new ByteArrayInputStream(fileContent));
String finalS3Url = s3Manager.getS3FileWebUrl(s3FilePath);
log.info("Test file successfully uploaded. Final URL: {}", finalS3Url);

Download file from storage

If the file on S3 has public read access rights then it can be easily downloaded by direct URL using the common approach to make HTTP Get request.

But if the file doesn't have public read access rights - it requires S3 credentials for downloading and should be downloaded by S3 API. For that you can use the following method from S3Manager:

public InputStream getFile(String bucket, String path)

Where:

bucket - is the target bucket name
path - path to target file including folders structure, file name and file extension, but without specifying the bucket name. E.g.: "folder1/subfolder/my_file.txt"

The following example demonstrates how to download text file from S3 and read its content:

String s3FilePath = "upload_file_test/example.txt";
InputStream fileContentInputStream = s3Manager.getFile("bucket", s3FilePath);
String fileContent = IOUtils.toString(fileContentInputStream, StandardCharsets.UTF_8);
log.info("Test file successfully downloaded. File content: {}", fileContent);

Delete file from storage

Method with the following signature should be used to delete file from S3:

public void deleteFile(String bucket, String path)

Where:

bucket - is the target bucket.
path - path to target file including folders structure, file name and file extension, but without specifying the bucket name. E.g.: "folder1/subfolder/my_file.txt"

Get list of files

Method with the following signature can be used to get the list of files:

public List<String> listFiles(String bucket, String path, String filter)

Where:

bucket - is the target bucket.
path - folder of folders structure of the files you're looking for. This parameter is used as prefix of file paths it searches. E.g.: "folder1/subfolder/", "folder2/".
filter - the Regexp pattern to filter the files. E.g. ".*\.txt"" to find any file with .txt extension.

The following example demonstrates how to get the list of all txt files:

String baseFolder = "upload_file_test/";
List<String> filesPathList = s3Manager.listFiles(baseFolder, ".*\\.txt");
for(String filePath: filesPathList) {
	log.info("The following file exist on S3: {}", filePath);
}

Complete example

UploadAndReadFilesFromS3.java

package eu.ibagroup.easyrpa.tests.task;

import eu.ibagroup.easyrpa.engine.annotation.AfterInit;
import eu.ibagroup.easyrpa.engine.annotation.ApTaskEntry;
import eu.ibagroup.easyrpa.engine.apflow.ApTask;
import eu.ibagroup.easyrpa.engine.service.StorageManager;
import eu.ibagroup.easyrpa.utils.IoUtils;
import lombok.SneakyThrows;
import lombok.extern.slf4j.Slf4j;

import javax.inject.Inject;
import java.io.ByteArrayInputStream;
import java.io.IOException;
import java.io.InputStream;
import java.util.List;
import java.util.UUID;

@ApTaskEntry(name = "Upload to S3")
@Slf4j
public class UploadAndReadFilesFromS3 extends ApTask {

	@Inject
	private StorageManager storageManager;

	private String baseFolder;

	private String s3FilePath;

	private byte[] fileContent;

	@AfterInit
	public void init() {
		String fileName = UUID.randomUUID().toString() + ".txt";
		baseFolder = "upload_file_test/";
		s3FilePath = baseFolder + fileName;
		fileContent = "Text file content example".getBytes();
	}

	@Override
	public void execute() throws IOException {
		uploadFile();
		printListFiles();
		readFile();
	}

	private void uploadFile() {
		storageManager.uploadFile(StorageManager.DEFAULT_S3_BUCKET, s3FilePath, new ByteArrayInputStream(fileContent));
		String finalS3Url = storageManager.getFileLink(StorageManager.DEFAULT_S3_BUCKET, s3FilePath);
		log.info("Test file successfully uploaded. Final URL: {}", finalS3Url);
	}

	private void printListFiles() {
		List<String> filesPathList = storageManager.listFiles(StorageManager.DEFAULT_S3_BUCKET, baseFolder, ".*\\.txt");
		for (String filePath : filesPathList) {
			log.info("The following file exist on S3: {}", filePath);
		}
	}

	@SneakyThrows
	private void readFile() {
		InputStream fileContentInputStream = storageManager.getFile(StorageManager.DEFAULT_S3_BUCKET, s3FilePath);
		String fileContent = IoUtils.toString(fileContentInputStream);
		log.info("Test file successfully downloaded. File content: {}", fileContent);
	}
}