OSG Document 1017-v1

High Throughput WAN Data Transfer with Hadoop-based Storage

Document #:
OSG-doc-1017-v1
Document type:
Paper
Submitted by:
Haifeng Pi
Updated by:
Haifeng Pi
Document Created:
18 Jan 2011, 14:51
Contents Revised:
18 Jan 2011, 14:51
DB Info Revised:
18 Jan 2011, 14:51
Viewable by:
  • Public document
Modifiable by:
Abstract:
Hadoop distributed file system (HDFS) is becoming more popular in recent years as a key building block of integrated grid storage solution in the field of scientific computing. Wide Area Network (WAN) data transfer is one of the important data operations for large high energy physics experiments to manage, share and process datasets of PetaBytes scale in a highly distributed grid computing environment. In this paper, we present the experience of high throughput WAN data transfer with HDFS-based Storage Element. Two protocols, GridFTP and fast data transfer (FDT), are used to characterize the network performance of WAN data transfer.
Files in Document:
Authors:
Associated with Events:
CHEP10 held on 18 Oct 2010 in Taipei, Taiwan
DocDB Home ]  [ Search ] [ Last 20 Days ] [ List Authors ] [ List Events ] [ List Topics ]

Supported by the National Science Foundation and the U.S. Department of Energy's Office of Science Contact Us | Site Map

DocDB Version 8.7.22, contact Document Database Administrators