The OPAL Discussion Forum

[Norman Huang <norman-huang AT hotmail.com>]

Hi, thanks for all the responses.

trcompdev.triumf.ca has 8 cores, and trcomp01.triumf.ca has 24 cores.

I ran a few more tests, and it seems to me like the effectiveness of the

number of cores depend on the grid size and possibly the number of

particles. For a 32x32x2048 grid with 50000 particles, the optimal

configuration was 4 cores. Specifically, a short tracking test produced

these results:

1 Core: 8:30min

2 Cores: 6:57min

4 Cores: 5:38min

8 Cores: 5:40min

16 Cores: 7:40min

Then I did the same test, but reduced the grid sizing to 16x16x16,

and I get:

1 Core: 3:00min

2 Cores: 4:12min

4 Cores: 8:01min

My guess is that the number of cores to use should scale according

to the calculation workload, if that makes sense?

Regards,

Norman 

 

> Date: Thu, 1 Dec 2011 08:55:52 +0100
> Subject: Re: [Opal] Re: Enabling Parallel Computing on OPAL-T
> From: christof.j.kraus AT gmail.com
> To: andreas.adelmann AT psi.ch
> CC: norman-huang AT hotmail.com; opal AT lists.psi.ch; raoyn AT triumf.ca; krab AT triumf.ca; tplanche AT triumf.ca; fwj AT triumf.ca
>
> Hi,
>
> suppose it were the interconnection, does this then make sense:
>
> * start cite Normans first email *******
> When it's being run, all 8 cpu cores are active, but only one is doing the
> calculations. Furthermore, this run actually took an hour more than a previous
> run using a single core
> * end cite Normans first email *********
> ?
>
> christof
> On Tue, Nov 29, 2011 at 7:03 AM, Andreas Adelmann
> <andreas.adelmann AT psi.ch> wrote:
> > Hello Norman
> >
> > I need to know how many cores trcompdev.triumf.ca has.
> > You are requesting 8 cores on one node (trcompdev.triumf.ca)
> > maybe you are simply overloading trcompdev.
> >
> > One other thing you have to be aware of is the interconnect bandwidth:
> > Gigabit will not work and you have in such a case the very same effect
> > that you are observing. Again Fred know all this. And YES the FFT based
> > solver
> > scales very well iff you have a fast interconnect.
> >
> > I also observer that you are using O(2E6) grid points with only 5E4
> > particles.
> > Your grid is far to large! Per grid point you should have on the average 10
> > ... 40
> > particles.
> >
> > My advice:
> >
> > a) check the how much steam you nodes has
> >
> > b) do you use a fast interconnect?
> >
> > c) reduce the size of the grid or increase the number of particles.
> > NP=5E4 -> G= 16x16x16
> >
> >  Cheers Andreas
> >
> >
> >
> > On Nov 28, 2011, at 10:40 PM, Norman Huang wrote:
> >
> > Hi Andreas,
> >
> > Here's the stdout output.
> >
> > Using this configuration, a run takes about 3.5 hours. While using a single
> > node only takes 2 hours.
> >
> > The Pi calculation test was successful, so I think OpenMPI is installed
> > correctly. Does FFT actually
> > exploit parallel processing, or do I have to use multigrid instead?
> >
> > Regards,
> > Norman
> >
> > ________________________________
> > Subject: Re: Enabling Parallel Computing on OPAL-T
> > From: andreas.adelmann AT psi.ch
> > Date: Sat, 26 Nov 2011 08:45:35 +0100
> > CC: opal AT lists.psi.ch; raoyn AT triumf.ca; krab AT triumf.ca; tplanche AT triumf.ca; fwj AT triumf.ca
> > To: norman-huang AT hotmail.com
> >
> > Hello Norman the input files is just fine, I made one small change not
> > related to your problem.
> >
> > Can you send me the std-output of your run? On which cluster do you run?
> >
> > Are you sure that MPI is installed properly i.e. can you run a canonical MPI
> > test - for example
> > calculating Pi - (this test comes with every MPI distribution).
> >
> > You should also contact Fred Jones, I remember having similar problems when
> > I was at Triumf
> > in the beginning of this year.
> >
> > Cheers Andreas
> >
> >
> >
> >
> >
> >
> >
> >
> > On Nov 26, 2011, at 2:54 AM, Norman Huang wrote:
> >
> > Hi Andreas,
> >
> > I'm trying to do space charge calculations using parallel computing on
> > OPAL-T, but
> > it appears only 1 core is being used.
> >
> > This is my fieldsolver parameters:
> >
> > Fs1:FIELDSOLVER, FSTYPE=fft, MX=32, MY=32, MT=2024,
> > PARFFTX=true, PARFFTY=true, PARFFTT=false,
> > BCFFTX=open, BCFFTY=open, BCFFTT=open,
> > BBOXINCR=1, GREENSF=INTEGRATED;
> >
> >
> > Command used to run the test 'isis.in': mpirun -np 8 opal --commlib mpi
> > isis.in
> >
> > When it's being run, all 8 cpu cores are active, but only one is doing the
> > calculations. Furthermore, this run actually took an hour more than a
> > previous
> > run using a single core.
> >
> > Am I missing some configurations?
> >
> > Regards,
> > Norman
> > <isis.in>
> >
> >
> > ------
> > Dr. sc. math. Andreas (Andy) Adelmann
> > Staff Scientist
> > Paul Scherrer Institut WLGB/132 CH-5232 Villigen PSI
> > Phone Office: xx41 56 310 42 33 Fax: xx41 56 310 31 91
> > Phone Home: xx41 62 891 91 44
> > -------------------------------------------------------
> > Thursdays: ETH CAB H 83.1  +41 44 632 36 72
> > ============================================
> > The more exotic, the more abstract the knowledge,
> > the more profound will be its consequences.
> > Leon Lederman
> > ============================================
> >
> >
> >
> > <stdout.txt>
> >
> >
> > ------
> > Dr. sc. math. Andreas (Andy) Adelmann
> > Staff Scientist
> > Paul Scherrer Institut WLGB/132 CH-5232 Villigen PSI
> > Phone Office: xx41 56 310 42 33 Fax: xx41 56 310 31 91
> > Phone Home: xx41 62 891 91 44
> > -------------------------------------------------------
> > Thursdays: ETH CAB H 83.1  +41 44 632 36 72
> > ============================================
> > The more exotic, the more abstract the knowledge,
> > the more profound will be its consequences.
> > Leon Lederman
> > ============================================
> >
> >
> >