An Open-source Based Data Management and Processing Framework on a Central Server for Electron Microscopy

The ever-expanding size of electron microscope (EM) data renders most data managements provided by proprietary software less efficient. Over the past few years several EM analysis packages have been developed and made available as open-source libraries. These libraries are continually growing and have provided elegant solutions to many issues. They have not yet, however, been consolidated into a standalone and open-source software suite capable of connecting users with the power of high performance computing (HPC) commonly provided by a central server at the research institutes. Despite a huge progress made by the community, organizing and processing the data in a streamline on the server still involves a steep learning curve and often proprietary software. An open-source high-level data management framework for all metadata, database for archiving, and visualizations of preliminary analyses is highly desirable. Euclid Techlabs, LLC submits this proposal to design an open-source data framework that is capable of managing, processing and analyzing EM-generated data. Our open-source framework aims at encapsulating the work needed by EM users for handling big data in an industrial best practice software implementable on a central computing server. It can dramatically reduce or completely lift the challenges off from control computers to perform data collection and conversion, run algorithmic analyses, archive and query in and from a database. Euclid has several unique advantages in this proposal: our tight collaborations with JEOL, BNL and NIST for hosting the tests needed; our R&D 100 award-winner TEM pulser product for cross- platform and multi-party data bridging; our team’s background in particle physics and accelerator physics data management and control software development for big data. Phase I merit and deliverable: We will build the main architecture of the framework which can be deployed first on a “mockup” central server at BNL and NIST, then on the HPC supercomputer at NERSC. We will be able to use the framework to automatically format, process, and visualize data from a JEOL microscopes at BNL and NIST. The framework deployed on the server should be able to respond to user requests for data manipulation and queries. The software proposed here focuses on bridging the gap between the EM community and the computing and storage power of servers at the labs. The software will be gladly accepted by both EM users and manufacturers because it allows for live data processing for big dataset generated by the advanced detectors. We will be able to attract interest from instrumentation vendors to share API for our framework to manage metadata produced by their hardware. We anticipate that with the hard work and influence of our collaboration, the community will open to adopting the open-source framework.