Read file from S3 bucket
Created: 2012-08-23 15:46:21
This workflow simply loads a text file that is stored in an AWS S3 bucket. It is provided as an example of how to do this, rather than be a complete, reusable solution. You need to have s3fs installed and configured with your AWS credentials to use this workflow (see http://code.google.com/p/s3fs/).
Preview
Run
Run this Workflow in the Taverna Workbench...
Workflow Components
Authors (1)
Titles (1)
Descriptions (1)
This workflow simply loads a text file that is stored in an AWS S3 bucket. It is provided as an example of how to do this, rather than be a complete, reusable solution. You need to have s3fs installed and configured with your AWS credentials to use this workflow (see http://code.google.com/p/s3fs/). |
Dependencies (0)
Inputs (3)
Name |
Description |
filename |
The name of the file in the bucket that you want to load. This filename should be relative to the root of the bucket, e.g. /my-file.txt or /path/to/my-file.txt
|
s3_bucket_name |
The name of the S3 bucket that you want to mount.
|
mount_point |
The local mount point in the filesystem to use. Note that this should be a directory that exists and that you have write access to (no trailing slash), e.g. /mnt/s3 or /media/cloud/my-bucket
|
Processors (4)
Name |
Type |
Description |
Read_Text_File |
localworker |
ScriptBufferedReader getReader (String fileUrl, String encoding) throws IOException {
InputStreamReader reader;
try {
if (encoding == null) {
reader = new FileReader(fileUrl);
} else {
reader = new InputStreamReader(new FileInputStream(fileUrl),encoding);
}
}
catch (FileNotFoundException e) {
// try a real URL instead
URL url = new URL(fileUrl);
if (encoding == null) {
reader = new InputStreamReader (url.openStream());
} else {
reader = new InputStreamReader (url.openStream(), encoding);
}
}
return new BufferedReader(reader);
}
StringBuffer sb = new StringBuffer(4000);
if (encoding == void) {
encoding = null;
}
BufferedReader in = getReader(fileurl, encoding);
String str;
String lineEnding = System.getProperty("line.separator");
while ((str = in.readLine()) != null) {
sb.append(str);
sb.append(lineEnding);
}
in.close();
filecontents = sb.toString();
|
mount_s3_bucket |
externaltool |
|
umount_s3_bucket |
externaltool |
|
build_full_file_path |
localworker |
Scriptoutput = string1 + string2; |
Datalinks (7)
Source |
Sink |
build_full_file_path:output |
Read_Text_File:fileurl |
s3_bucket_name |
mount_s3_bucket:bucket_name |
mount_point |
mount_s3_bucket:mount_point |
mount_point |
umount_s3_bucket:mount_point |
filename |
build_full_file_path:string2 |
mount_point |
build_full_file_path:string1 |
Read_Text_File:filecontents |
text |
Coordinations (2)
Controller |
Target |
Read_Text_File |
umount_s3_bucket |
mount_s3_bucket |
Read_Text_File |
Uploader
License
All versions of this Workflow are
licensed under:
Version 1
(of 1)
Credits (1)
(People/Groups)
Attributions (0)
(Workflows/Files)
None
Shared with Groups (1)
Featured In Packs (0)
None
Log in to add to one of your Packs
Attributed By (1)
(Workflows/Files)
Favourited By (0)
No one
Statistics
Other workflows that use similar services
(0)
There are no workflows in myExperiment that use similar services to this Workflow.
Comments (0)
No comments yet
Log in to make a comment