Cosmic Ray e-Lab: Fellows resources

Cosmic Ray Introduction - Catholic University Aug 2021

Submitted by MarkAdams on Mon, 08/16/2021 - 11:01

Slides from an introduction to cosmic rays and how teachers can observe them using QuarkNet detectors. Talk by Mark Adams at the QuarkNet summer workshop at Catholic University August 16, 2021.

Read more about Cosmic Ray Introduction - Catholic University Aug 2021

Resources for Cosmic Ray Analyses Online

Submitted by MarkAdams on Tue, 03/31/2020 - 16:46

Cosmic Ray Analysis Opportunities

Read more about Resources for Cosmic Ray Analyses Online

Improvements to CR e-Lab

Submitted by kcecire on Wed, 02/05/2014 - 10:42

We propose a meeting at FNAL to discuss the e-Labs (31 March- 2 April). While most of our concerns are about Cosmic, we may also discuss other e-Labs for a limited time.

Our experience with phone- and video-conferences is limited to one- or two-hour events. Even these are sometimes hard to maintain connections and focus with all attendees. This meeting will take place over three days. We feel that this duration is difficult with participants in different places.

We suggest the following attendess: Bob, Liz, Edit, Tom. Ken might also wish to come for the Cosmic discussion. We can set-up a two-hour conference call with Tom McCauley and Dale Ingram near the end of our time.

DIscussing these issues will help us to decide how best to spend our limited resources. We have been squashing bugs and putting out fires for too long. It's time to look at the entire system critically and decide what we can afford to address and what we have to leave as it is.

JUST IN: We might need to take into account that we will be working with Drupal....

Abbreviated List (March 26)

Watch standing
Who is watching the servers to be sure that everything is up. Who can they call if something goes astray?

Daily server observation
What does Edit do during in the background to keep an eye on the servers? Is nagios enough?

Rollout and Rollback instructions

Do we have clear instructions about how to do these?

Something bad is happening

What do we do when things go astray? When is it OK to restart the server?

Drupal accounts

Who will approve these new accounts as they come in?

Remains from original list

Concurrency testing
Load balancing (priority queue)
Data uploads
Remove bottlenecks in searching
Bugs
Feature requests

Original Long List

Topics for discussion include:

Performance of the Cosmic e-Lab

The e-Lab is showing its age. It falls over too often in workshop or classroom settings. We need to explore ways to address this. We have just rolled out a big change in analysis workflows that should help with performance issues. We plan to test this recent update, but need to also discuss other areas where we can improve performance. These inlcude:

re-plotting
- We currently re-run an analysis if the user wants to do something as re-scale the axes. This is a waste of compute cycles. The bless plots allow for redoing axes just by clicking on the plot. We should add that feature to plots from the analyses as well.
  - EP: This could take some time because we need to change the graphics tool from svg to flot. That involves changing the perl code, possibly the swift scripts and new coding for flot, json and gson.
Java vs. Perl
- All of the analysis workflows rely on aged Perl scripts. We moved ThresholdTimes to Java and realized a big performance boost. Is it worthwhile to consider re-coding the rest of the workflows in Java?
  - EP: That could be done but it will take some time to convert from the Perl to the Java plus testing to make sure that all works well (including rounding). Also we would need to modify swift scripts to call java rather than perl (it might be easy, not sure).
Concurrency
- The e-Lab gets very unresponsive when there are multiple users. This problem may have improved with the persistence of the .thresh files. We should still consider testing before the summer workshops start.
  - We might try running new stress tests with multiple users and see if this is still the problem. The last couple of reports that I received about the server "hanging" were due to other reasons (still investigating that but not due to multiple users/analysis running).
Load-balancing of jobs
- Is it possible to make a queue that will manage jobs in order to reduce pile-up?
  - EP: I am working with this and we will soon have a queue for testing :)
problems with scaleability - too many data files plus a more complicated search with blessing are bogging down searches (do we need to keep all of these files? Can we simplify what we search for?)
- We need to discuss this in a telecon and come up with ideas by looking at the problem areas (cosmic search data).
review our infrastructure (vdc, physical files, etc).
- We probably need to involve other team members to discuss this issue (Mihael, Mike W.).
Workshop use is limited because response time is so long when 15 students make simultaneous requests. Improvements in multiple job submission or a list of conditions that allow 10 simultaneous jobs would be helpful.
- We need to test this again.
Data uploads
- Is there any way to improve the speed of data uploads? They take so long that something must be wrong on the server side. Tom sent mail about this on 19 Feb.
  - We need to find tools to test what causes the delay. Uploading from the user machine to the server can some time depending on the size of the file, the internet connection (I guess), browsers (?), etc.

Look and feel

The e-Lab's user interface (UI) is also showing its age. There may be easily-accessible (i.e., free or cheap) solutions to updating the current UI codebase to something that looks and feels like 2014. There are also specific interface issues beyond this large picture. These include:

Shower interface
- When doing shower analyses users should be able to look for nearby detectors. The current interface gives them no information about separation. Users could select a detector at Fermilab and one in Japan. We've talked about using a map and a circle, but it doesn't have to be that fancy.
  - This problem could be solved now without coding for a tool.
Can we use mashups to make 2014 UI? Could this improve how we display logbooks, milestones and references?
- Needs to be well-thought out. Not difficult to implement but time consuming because you have to work with the whole framework.
A better interface for the registration and management of users
- This functionality needs a major rewrite. Not difficult, time consuming.

Data mismatch

Location mismatch
- if a teacher has an account at a school in Wisconsin and uploads data, the data looks like it is in Wisconsin. If that teacher takes the detector somewhere else (e.g., the South Pole) and uploads data, the data look like they are in Wisconsin.
  - Add documentation for teachers to get new accounts if they move schools or want to take data far away from their school
  - Getting a new account is how we currently do this. It's a workaround and puts the burden on the user. The difficulty comes from the metadata for the location of the uploaded data comes from the teacher's loging. The metadata could also come from the geometry of the detector. This is a much bigger solutio and harder to code--maybe it's "pie in the sky," but it solves, rather than works-around, the problem. Doing location this way also makes it easier to change the shower interface.

Keeping up with new versions of system level software

We need to keep on the web treadmill as our software stack ages.

Java 6 vs 7 / Tomcat 6 vs 7
- Already working with this.
- We need to upgrade. This includes setting up the environment and make sure our existing code migrates OK

Unit Testing

The webapp lacked unit testing: code that is used to determine that each part of a program is working correctly. These tests are written at the same time the code is written to prevent buggy code and later to make sure that the code is still working with updates, new additions to the code, etc. Started adding thse but it is still a working in progress.
Also, we need to add some automatic testing: the deployment of the application has presented some problems a few times. It seems that there is a race of threads at some point and the compilation of some classes is not 100%. A new deployment usually fixes this but we need some automated tools to tell us whether the rollout is 100% instead of manually testing the website.

Better Administrative Tools

More monitoring tools, etc.
- Replicate command line ways of retrieving information from the server
- Make sure that the statistics we are gathering are correct (e.g., number of pretests
- A tool like NGOP at Fermilab that checks whether the site is up. It emails or calls a telephone if the site is down. Argonne must have this.

Documentation for Project

What can we use to store all the information about the system?
University of Chicago Developer's wiki vs Drupal.
Do we need more documentation on the analyses and how they work?

Scaffolding for Users

Should we update our screencasts or add more? (e.g., screencast of data blessing on upload)
Update the references associated with milestones.

Understanding our users and e-Lab usage

Inspect the Tomcat log for feedback (analyses, cms event display, interactions with the database).
Look at trends in the number of logbook entries, pre and post-tests entries and other items in the e-Lab over time.
Use a tool like Google analytics to see what the users are doing (logbook, milestones, references, etc.?)
Generate a survey to learn what the users (and not users) are doing.

Marketing our e-Labs

How can we retain and increase our audience?
Make NextGen more prominent.

Examination of the original assumptions and goals of project, how we have succeeded and whether they are still relevant.

(e.g., sharing data especially for shower studies, providing analysis tools to schools)
Is a shower study using multi-school data practical with our tools? Do we need more functionality and visualization?
Do we need the tight coupling between milestones and the logbook?

How does the Purdue Java tool influence what should be in the e-Lab?

What new technologies might replace what we have implemented

Should we be replacing log books with some open source solution.
Should we be using Cloud Computing?
Should we still be using Tomcat and Java Server Pages?

Getting rid of wiki dependency

Handling glossary, etc. consistency in all e-Labs

Should we continue to try to support and develop the CMS and LIGO e-Lab?

Yes. Next question.

Hardware Needs

Feature Requests

allow user to superimpose multiple data sets on one plot, i.e., Barometric Pressure vs. Flux.
histograms of times between counters "i" and "j" (6 possible 2-fold combinations) Aplications in order of usefulness: speed of muon; performance studies; calibration of relative counter timing for use in shower reconstruction and for estimates of intime counter hits versus random backgrounds.
Counter multiplicity and logic requirements for any analysis. Install ability to require counter a specific set of counters, e.g. Counters 1 and 3 but not counter 4, within a user-specified time window. This is critical in lifetime studies. If relative times from suggestion 1 were available per event, that would also enhance muon lifetime measurements.

Read more about Improvements to CR e-Lab

Timing From Raw Data

Submitted by rspete on Mon, 10/08/2012 - 15:17

Timing From Raw Data

A brief attempt at explaining a time extraction from raw data.

Sample Time Calc

QDAQ.exe

A windows-based program has been developed that will allow one to review a raw text data file, as well as to translate it into a form that can be analyzed using Excel.

Features

Read more about Timing From Raw Data

Estimating Flux Rates

Submitted by rspete on Mon, 10/08/2012 - 15:15

Estimating Flux Rates

Muon Flux rates are strongly affected by: Scintillator surface area Orientation Spacing and solid angle acceptance Intervening absorber material and thickness Altitude

A simple model is presented to allow estimating flux rates due to area, spacing, and absorber.

Muon Rate Estimator

Read more about Estimating Flux Rates

Blessing Proposal Discussion

Submitted by rspete on Mon, 10/08/2012 - 15:05

Blessing Proposal Discussion

Proposal for Blessing data, July 2007.

Media:Blessing+proposal.doc

Article from Auger group on Blessing data. [1]

Read more about Blessing Proposal Discussion

Planning an experiment

Submitted by rspete on Mon, 10/08/2012 - 14:55

Planning an experiment

This is an Excel spreadsheet template for calculating how long you might need to run an experiment to see statistically significant differences in count rates.

Read more about Planning an experiment

Running A Workshop

Submitted by rspete on Mon, 10/08/2012 - 14:40

Running a workshop

(Redirected from Running A Workshop)

This file was made during the leadership session at Fermi Lab on Sunday, 7/15. Please improve and delete this disclaimer.

Workshop Checklist

Read more about Running A Workshop

Plateauing Detectors

Submitted by rspete on Mon, 10/08/2012 - 14:32

Plateauing Detectors

"6000" CRMD Counters, Current version: Nov 2011

Remade from previous versions; tested by Jeremy Paschke and Martin Shaffer, Summer 2009

Students will require both following files to plateau.

6000 HOWTO Plateauing Powerpoint

6000 Plateauing Spreadsheet Form

Older, out-of-date versions

From Jeremy Paschke, Summer 2008

These two documents include edits and remakes of Fellows documents listed above. Fixes fundamental errors. Thanks to Jeremy. Fellows, please review and add edits. When all changes have been received, these will be moved into the Cosmic e-Lab proper for public use.

HOWTO Plateauing Powerpoint: How-to-Plateau-2008JP

Plateauing Spreadsheet Form: Plateau-Template-2008JP

From Fellows, Summer 2007

We've got a nice powerpoint on connecting all the equipment and plateauing the counters. The level is intended for beginning students or teachers and is designed to march through the procedure as time efficiently as possible, so that anyone can start to do real science asap.

HOWTO Plateauing Powerpoint: How+to+Plateau

There's an excel spreadsheet template associated with these directions. The powerpoint has a link to the spreadsheet, but you may have to open it by yourself if you download from here.

Plateauing Spreadsheet Form: Plateau+Template