1. Intriduction to Java Programming for Beginners, Novices ...

1. Intriduction to Java Programming for Beginners, Novices ...

Parallel Programming Dr Andy Evans A few terms from standard programming Process: a self-contained chunk of code running in its own allocated environment. Thread: a lightweight sub-process; a semi-independent chunk of a program separated off from other elements. Processor: chip doing processing. One Processor may have multiple Cores. A PC might have multiple Central Processing Units (~processor plus other bits), but will undoubtedly have multiple Cores these days. Core: a processing unit usually only capable of running a single Process at a time (though can have

others on hold). Usually a single core machine can appear to run more than one Process by quickly switching between processes, though more recently have multiple Hardware Threads (HW Threads) to support effective use and/or multiple processes/threads essentially as virtual cores. Asynchronous programming: calling code that will, at some point get back to us, rather than calling it in strict order. Concurrent programming: programming using multiple threads and/or multiple cores; on a single machine or multiple specialised machines. How to check processor/core numbers My Computer Properties Right-click taskbar Start

Task Manager ( Resource Monitor in Win 8) With Python: multiprocessing.cpu_count() Computational issues with modelling High Performance Computing Parallel programming Distributed computing architectures

The frontier of modelling Individual level modelling is now commonplace. Data is in excess, including individual-level data. Network speeds are fast. Storage is next to free. So, what is stopping us building a model of everyone/thing in the world? Memory. Processing power. Memory To model with any reasonable speed, we need to use RAM.

Sex: 1bit (0 = male; 1 = female) 1 bit = 1 person 1 byte = 8 people 1Kb = 1024 x 8 = 8192 people 1Mb = 1,048,576 x 8 = 8,388,608 (10242x8) people 1 Gb = 1,073,741,824 x 8 = 8,589,934,592 people Seems reasonable then. Typical models running on a PC have access to ~ a gigabyte of RAM memory. Memory Geographical location ( N &W): 8 ints = (for example) 256 bits N &W): 8 ints = (for example) 256 bits

1 Gb = 33,554,432 people This isnt including: a) The fact that we need multiple values per person. b) That we need to store the running code. Maximum agents for a PC ~ 100,000 1,000,000. Processing Models vary greatly in the processing they require. a) Individual level model of 273 burglars searching 30000 houses in Leeds over 30 days takes 20hrs. b) Aphid migration model of 750,000 aphids takes 12 days to run them out of a

100m field. These, again, seem ok. Processing However, in general models need multiple runs. Models tend to be stochastic: include a random element so need multiple runs to give a probabilistic distribution as a result. Errors in inputs mean you need a distribution of inputs to give a reasonable idea of likely range of model outputs in the face of these errors. Monte Carlo testing

Where inputs have a distribution (error or otherwise), sample from this using Monte Carlo sampling: Sample such that the likelihood of getting a value is equal to its likelihood in the original distribution. Run the model until the results distribution is clear. Estimates of how many runs are necessary run from 100 to 1000s. Identifiability In addition, it may be that multiple sets of parameters would give a model that

matched the calibration data well, but gave varying predictive results. Whether we can identify the true parameters from the data is known as the identifiability problem. Discovering what these parameters are is the inverse problem. If we cant identify the true parameter sets, we may want to Monte Carlo test the distribution of potential parameter sets to show the range of potential solutions. Equifinality In addition, we may not trust the model form because multiple models give the same calibration results (the equifinality problem). We may want to test multiple model forms against each other and pick the best. Or we may want to combine the results if we think different system components

are better represented by different models. Some evidence that such ensemble models do better. Processing a) Individual level model of 273 burglars searching 30000 houses in Leeds over 30 days takes 20hrs. 100 runs = 83.3 days b) Aphid migration model of 750,000 aphids takes 12 days to run them out of a 100m field. 100 runs = 3.2 years Ideally, models based on current data would run faster than reality to make

predictions useful! Issues Models can therefore be: Memory limited. Processing limited. Both. Solutions If a single model takes 20hrs to run and we need to run 100:

a) Somehow cut down the number of runs needed. b) Batch distribution: Run models on 100 computers, one model per computer. Each model takes 20hrs. Only suitable where not memory limited. c) Parallelisation: Spread the model across multiple computers so it only takes 12mins to run, and run it 100 times. Computational issues with modelling High Performance Computing Parallel programming Distributed computing architectures

Supercomputers vs. Distributed Supercomputers: very high specification machines. Added multiple processors to a single machine with high speed internal connections. Note that most PCs now have more than one processor and/or core. Distributed computing: Several computers work together. Either formally connected or through apps that work in the background. Strictly includes any networked computing jobs including Peer-to-Peer (P2P) services. Informal includes: Napster (Distributed Data); [email protected] (Distributed Processing; see Berkeley Open Infrastructure for Network Computing [BOINC]);

Skype; Spotify. Flynns taxonomy SISD: Single Instruction, Single Data stream MISD: Multiple Instruction, Single Data stream SIMD: Single Instruction, Multiple Data stream Each processor runs the same instruction on multiple datasets. Each processor waits for all to finish. MIMD: Multiple Instruction, Multiple data stream Each processor runs whatever instructions it likes on multiple data streams. SPMD: Single Process/program, Multiple Data

Tasks split with different input data. Beowulf Formal MIMD architectures include Beowulf clusters. Built from cheap PCs, these revolutionised the cost of HPC. Generally one PC with a monitor acts as node zero collating and displaying results. Other nodes can write to their own drives and a network space (Shared Memory Model). Parallelisation

Split the model up so bits of it run on different machines. End result then collated. Two broad methods of parallelisation which play out in Flynns taxonomy, but also at the model level: Data parallelisation Divide the data the model works with into chunks, each processor dealing with a separate chunk (in our case, we usually divide the geography up). Task parallelisation Each processor has all the data, but the task is split up (in the case of an ABM, the agents might be divided up though whether this is task or data division depends on the agents).

Which? If memory limited, you have to divide the memory-heavy components, even if this slows the model. Sometimes it is better to get a model running slowly than not at all. Otherwise, whichever reduces communication between processors this is usually the slowest process. If agents local and static, then divide geography. If agents move lots but dont communicate, then divide agents. Unfortunately, most models have agents that move and communicate so at some point youll have to move agents between geography slabs or communicate with agents on other nodes.

Case Study Sometimes you need to think closely about the data transferred to get out of this issue. Memory limited model: how to model millions of Aphids attacking agricultural land? Aphids move a mix of long and short distances (Lvy flight), random but skewed by wind. Long flights take place when density of aphids are high, so we cant reduce the number of agents. i.e. model needs all of geography on one node, but also all agents need to know

about all other agents (i.e. communicate with other agents). Seems problematic. Case Study Lets say we run the model on 10 nodes, each with the whole geography but we split up the aphids. We might think that 100 aphids need 99 communications each to find out where all the other aphids are (i.e. 9,900 communications per step). But, actually, they only need the density raster on each node. i.e. at most, each node needs to communicate with each other node once per step (10 x 9 communications). Actually, if we get node zero to request and send out the total aggregate density, each

node only needs to communicate with node zero (i.e. 10 sends and 10 receives). Managed to model 1 million aphids at an equivalent speed to 100,000 aphids on one processor. Issues with parallelisation Message passing overheads if we need our machines to talk. Need to lock shared data when being altered. This can result in thundering herd problems, where a number of processes want access to resources and use up processing power negotiating that and then putting non-successful processes to sleep. Need to carefully plan shared variables to prevent race hazards, where the order

of variable changes determines their proper use. Load balancing (how to most efficiently distribute the processing and data). Synchronisation/Asynchronisation of code timings to avoid detrimental blocking (one free processor waiting on another), particularly deadlock (where all the processors are waiting for each other). Computational issues with modelling High Performance Computing Parallel programming

Distributed computing architectures Threads Thread: a lightweight sub-process; a semi-independent chunk of a program separated off from other elements. Often a system will allocate different threads to different CPU cores to maximise efficient use of the cores. We have seen that we can identify the "main" thread of a Python program with: if __name__ == '__main__': # code to only run in the main thread when run as a program

# not, for example, when imported as a module Python can utilise threads. Python Threads from threading import Thread def func(a): print(a) thread1 = Thread(target=func, kwargs={'a':10}) thread2 = Thread(target=func, kwargs={'a':20})

thread1.start() thread2.start() thread1.join() # Wait until thread1 finished thread2.join() Python Threads Threading works ok for IO, but references to variables reveal issues. In Python, objects are shared between all threads, rather than associated with them each. Standard Python includes something called the Global Interpreter Lock (GIL). This is used to ensure that only one thread at a time can use the Python objects. Early on, this seemed like a good idea: it means, for example, one thread can't remove the last

reference to a variable that another might be using. Unfortunately, it means that even where threads are allocated to different cores, they are still largely tied together - threads can't head off and do their own thing faster, as they can in other languages, as they have to wait while they all negotiate access to the objects with the GIL. Python Threads Even with the GIL, standard objects are not well set up to work with threads. There is, for example, the potential for multiple threads to try and adjust the value of an object in some confused sequence. Some objects are specially set up to work with threads ("thread safe" objects). These

can be accessed by multiple threads simultaneously (often "shared memory" objects) They are often kept safe from being adjusted by multiple threads simultaneously by being locked so only one thread can access them (the threads/objects are "synchronised"), and are generally set up to behave better when manipulated by multiple threads. For example, a queue.Queue can be passed into several threaded functions, and things added from each thread. from threading import Thread from queue import Queue

(very) Simple queue example def func(a, q): q.put(a) qu = Queue() thread1 = Thread(target=func, kwargs={'a':10, 'q': qu}) thread2 = Thread(target=func, kwargs={'a':20, 'q': qu}) thread1.start() thread2.start() thread1.join()

thread2.join() for item in range(qu.qsize()): print(qu.get()) Python Threads Unfortunately, even with thread safe objects, threads don't work well in Python. Any attempt to use variables generally increases the time involved as the GIL comes into play. Note that there are some exceptions. Some libraries, that call on C code beneath Python have their own C-based threading which works well (numpy, for example). If you need to worry about this (for example, because you have to

build the code to exploit this), details will be in the documentation. Multi-processor programs The solution to the threading issue is to run multiple Python interpreters, each running a separate process. Objects can be transferred between processes by "pickling" them - that is, serialising them into transferable binary file objects using Pythons "pickle" module. For a single machine with multiple cores, this is done invisibly by the standard library concurrent.futures module. This consists mainly of process Executors, and Futures which represent running processes and their potential ability to return results.

Multi-process programs import concurrent.futures def func(a): print(a) if __name__ == '__main__': ppe = concurrent.futures.ProcessPoolExecutor(max_workers=4) ppe.submit(func,"hello world") ppe.shutdown() import concurrent.futures

Getting results def func(a): print(a) if __name__ == '__main__': ppe = concurrent.futures.ProcessPoolExecutor(max_workers=4) args = ["10","20","30","40"] future_set = {ppe.submit(func, arg): arg for arg in args} done, not_done = concurrent.futures.wait( fs= future_set,timeout=None,return_when=concurrent.futures.ALL_COMPLETED)

for future in future_set: print(future.result()) ppe.shutdown() import concurrent.futures Getting results def func(a): print(a) if __name__ == '__main__':

ppe = concurrent.futures.ProcessPoolExecutor(max_workers=4) args = ["10","20","30","40"] results = {ppe.submit(func, arg): arg for arg in args} for future in concurrent.futures.as_completed(results, timeout=None): print(future.result()) ppe.shutdown() Multi-process programs import concurrent.futures def func(a): print(a)

if __name__ == '__main__': ppe = concurrent.futures.ProcessPoolExecutor(max_workers=4) args = ["10","20","30","40"] ppe.map(func,args) ppe.shutdown() Other functions Can also check whether tasks are running or not, and cancel them. future.add_done_callback(fn) Allows you to add a callback function that is called and passed the future object when the task is done. You can then extract the result from the

future object as before. The library can also be adjusted easily to make the program run as threads rather than processes, meaning it is simple to develop threaded programs where efficient, but convert them when it becomes less so. Distributed parallelisation If we want to do the same thing across multiple machines, the best way is to use celery: http://www.celeryproject.org/ This sets up: A broker: negotiates everything. A backend data store: stores results.

Worker machines: that do the jobs. Largely for batch distribution of jobs, but works within programs, for example to batch distribute function calls. Broker and Backend Broker negotiates which machine is to do which job. Common example is RabbitMQ: https://www.rabbitmq.com/ This uses the binary-level Advanced Message Queuing Protocol (AMQP) to transfer messages between machines. Backend stores results. Common example is Redis: https://redis.io/ but you can also use

RabbitMQ. Workers register with the broker and backend, and then can be allocated tasks. import celery app = celery.Celery('cel', broker='amqp://FIRSTPC', backend='redis://SECONDPC') @app.task def func(a): return "a = " + a

if __name__ == '__main__': args = ["10","20","30","40"] results = [func.delay(arg): for arg in args] for result in results: print(result.get()) Example: cel.py How to 1. Start the broker on FIRSTPC. 2. Start the backend on SECONDPC.

3. Start cel.py on THIRDPC (or more): celery -A cel worker --loglevel=info This will start as many workers containing the func as cores available. 4.Run the cel.py as a Python program on FOURTHPC. Each call to func from the program running on FOURTHPC will use one of the workers on THIRDPC, and the results will be retrieved from the backend on SECONDPC. Other elements Supervisor: A Process Control System (for POSIX systems): allows you to control the worker and other processes.

http://supervisord.org/ Flower: a celery control system: http://flower.readthedocs.io/en/latest/ Includes controlling tasks to workers, number of workers, task completion and stats, Github integration. (See also jobtastic: https://policystat.github.io/jobtastic/ especially for outputting reports) Downsides There's obviously an overhead associated with making and retrieving the results.

The more calls, the more overhead. Hard to do direct machine-to-machine communication. For complete communication control, you need MPI, the Message Passing Interface. Parallel programming Various options, but a popular one is the Message Passing Interface (MPI). This is a standard for talking between nodes implemented in a variety of languages. With shared memory systems, we could just write to that, but enacting events around continually checking memory isnt very efficient. Message passing better. A good implementation for Python is SciPy's MPI4Py http://mpi4py.scipy.org/docs/usrman/index.html

Optimised for transferring strings, 1D numeric arrays, and numpy-style data objects, all of which ~generally occupy contiguous memory (single-segment buffer interface). Runs on top of standard versions of MPI, so one of those must be installed and set up. Popular is https://www.open-mpi.org/ General outline You write the same code for all nodes. However, the behaviour changes depending on the node number. You can also open sockets to other nodes and send them stuff if they are listening. if (node == 0):

listen() else: sendData() Usually the MPI environment will organise running the code on the other nodes if you tell it to run the code and how many nodes you want. MPI basics API definition for communicating between Nodes. comm = MPI.COMM_WORLD Get MPI reference

comm.Get_size() Get the number of available nodes. comm.Get_rank() Get the node the code is running on. Load balancing This kind of thing is common: node_number_of_agents = 0; if (node != 0): node_number_of_agents = number_of_agents /(number_of_nodes - 1) if (node == (number_of_nodes 1)):

node_number_of_agents = node_number_of_agents + (number_of_agents % (number_of_nodes - 1)) agents = []; for i in range(node_number_of_agents): agents.append(Agent()) Sending stuff node_to_send_to = 1 message_ID = 10 If object is a pickleable object: comm.send (data, dest=node_to_send_to, tag=message_ID)

If object is a numpy array etc.: comm.Send([data, MPI.INT], dest=node_to_send_to,tag=message_ID) (For other data types, see https://mpi4py.scipy.org/docs/apiref/mpi4py.MPImodule.html) Receiving stuff data = comm.recv(source=0, tag=10) Might, for example, be in a loop that increments nodeSending, to recv from all nodes.

For numpy arrays etc., make the array first for maximum efficiency: data = numpy.empty(1000, dtype='i') comm.Recv([data, MPI.INT], source=0, tag=10) MPI Overall, MPI is not so hard to use. However, the earlier you start thinking about it, the easier it is. If you think a task requires full parallelisation, if you can build this in from the start, it is simpler than hitting a processing wall on a single thread program, and retro-fitting parallelisation.

multiprocessing library This is in the standard library, essentially following the threading library. More complicated to use than concurrent.futures. Allows for multi-processor and multi-machine use, including setting up pools of workers, but lacks the enterprise-oriented structure of celery. Can work with shared memory objects or message passing, but less standardised than MPI. Nevertheless, built into the standard library, making it easy to use, and worth knowing about. https://docs.python.org/3/library/multiprocessing.html

Summary We've seen some different approaches: 1. Threading (good where the limit is some other external code). 2. Concurrent use of cores: good for speedup where cores available, but doesn't solve memory issues. 3. Distributed batch tasks with celery. 4. Full parallelisation with MPI. 5. Multiprocessing library offers a broad toolkit. Computational issues with modelling High Performance Computing

Parallel programming Distributed computing architectures Issues with architecture Is there going to be a lot of communication? Can you cope with security issues? What skills do you need? Do you have the computing resources?

What other services do you want? Do you want a permanent resource? Communication and Processing speed Different computing components have different speeds: Central Processing Units can now process >7000 MIps Typical RAM read speeds are ~3000 Mbps. Typical hard-drive reading speeds are 700 Mbps. Hence we dont want to read hard-drives, and RAM speed limits us. However, what limits local computation is bus speeds: Typical System Bus transfer rates are ~1000 Mbps.

Typical IO Bus for hard-drives run at 133 Mbps. Latency and Location However, distributed computing relies on network speeds, or bandwidth. Theoretical values, however, are altered by the processing time needed for management, and sometimes by the distance and network form between exchanges. This gives us the network latency the speed it generally works at. Latency and Location Typical home network runs at 1.6Mbps. Typical Ethernet connection on a Local Area Network (LAN) runs at 10Mbps.

Typical fast Ethernet runs at 100Mbps. i.e. at best the same as hard-drive access. We therefore want to minimise computer-to-computer communications and minimise the distance between computers, ideally ensuring they are all on a Fast Ethernet LAN. Speedup One would expect that doubling the processors would halve the time. However, as Amdahl's law points out, this is limited by the speed of the nonparallelisable component, and this is particularly key in locking algorithms and those with high communication overheads. In general, parallelisation doesnt speed up models.

Infact, if we use communication across high-latency connections, there can be a slow-down in processing. We therefore generally parallelise models to make them possible, not faster. Security In general MPI-style coding allows outside code to contact each PC and run arbitrary Java. This needs a good firewall around, but not between, the PCs with strong security measures. Generally, with Beowulf setups, the machine-to-machine communications are encrypted and validated using Secure Shell (SSH), because Beowulf machines tend

to use the LINUX OS: http://en.wikipedia.org/wiki/Secure_Shell But it depends on your software, MPJ Express for Windows, for example, relies more on an external firewalls. Skills Other than MPJ Express, a lot of these systems run on Unix-like OSs like Linux. Useful to get familiar with these. Command line driven, but with various different shells on the same machine. Tend not to have lettered hard-drives, but instead space mounted as directories.

Learning: Mac-OS is a Unix-based system, and you can access the command line using the Terminal app. http://www.virtualbox.org/ allows you to run Linux on a PC. Linux Books Richard Petersen (2008) Linux: The Complete Reference. Generally a good starting point. Emmett Dulaney (2010) Linux All-in-One For Dummies.

Includes LAN and security setup. Basic tutorial at: http://www.ee.surrey.ac.uk/Teaching/Unix/ Volunteer computing Most fully Peer-to-Peer software is written bespoke and not so useful for processing as need a central node to report to. Easiest option for more centralised distribution is the Berkeley Open Infrastructure for Network Computing (BOINC): http://boinc.berkeley.edu/trac/wiki/ProjectMain BOINC client fetches jobs from a server and runs it on a local application.

It then returns the result. Client runs as a screensaver or on spare CPU cycles. Volunteer computing Large numbers of computers at low hardware cost (+ low maintenance etc.) High latency, so low communication/data transfer, high processing, jobs good. Person investment high as needs to have good looking interface and run reliably. BOINC suggest ~3 person-months: 1 month experienced sys admin; 1 month of a programmer; 1 month of a web developer

+ then 50% person to maintain it over project lifetime. Multi-core machines / GPUs It's possible now to buy multicore machines with large amounts of memory (generally Linux machines, as traditionally Windows machines had memory maximums). However, these can be expensive. Increasingly common is to instead use graphics cards. Graphics Processing Units generally have much more power and many more cores (often hundreds) for a very cheap cost.

GPUs The downside is that software needs writing in special languages, for example OpenCL. One option is Theano (comes with Anaconda), a compiler that will compile Python maths and array operations especially for GPUs such that they can be called very efficiently from standard Python. http://deeplearning.net/software/theano/ If you have a graphics card (or more than one - you can build banks of them) and need to do intensive array/matrix operations, this is a good option.

Beowulf In general, while wed distinguish Beowulf by being a cluster of PCs dedicated to parallelisation surrounded by a specific firewall, theres little difference between that and a Windows cluster running MPJ (though you can run MPJ on much more sophisticated architectures). Beowulf clusters have the great advantage of being cheap, easy to set up, and under local control. They are also on a LAN. You need to buy the PCs though, and make sure of their security and management. Limited in other resources they connect to.

Grid Computing More general than Beowulf (includes some things like BOINC and web-services), but tends in practice to be a formal architecture. A group of networked resources, including data servers, service providers, secure gateways, etc. managed by a consortium. Jobs timetabled/allocated to processors using middleware, e.g. the Globus Toolkit. Makes batch distribution simple: just load up the model on multiple processors. You can then have a single program that collates the end results. Grid

Generally maintained and secured by a consortium who own the machines. Low(ish) cost of entry. Good connectivity with resources. Share processing/memory with other people, so you need to wait for space to run stuff. Running on The Grid Because GRID's are shared between multiple users, they use 'job submission' systems. You submit your program to a queue and wait your turn. The larger the job (in terms of number of cores and amount of memory requested) the longer you usually have to wait.

Although it is possible to ask for an interactive session, it is normal to write a script to define the job. Each user has a resource limit (e.g. total number of CPU time). If you go over this you have to ask for / pay for more time. (Using the Leeds grid 'Arc2' is free for end-users). For more information about getting access to the GRID at Leeds, email Andy or Nick Malleson. Cloud computing Large scale processor farms with associated data storage and services. You rent as much power and space as you need elastically.

Popular versions include Amazon Elastic Compute Cloud (Amazon EC2) : http://aws.amazon.com/ec2/ Usually get a virtual machine you can work with (e.g. Amazon Machine Image (AMI) system). This may include virtual clusters for HPC: http://aws.amazon.com/hpc-applications/ Nice video at: http://www.youtube.com/embed/YfCgK1bmCjw Typical Amazon costs for Linux (Windows a bit more): Small (Default) $0.090 per Hour. Costs

1.7 GB memory 1 EC2 Compute Unit (1 virtual core with 1 EC2 Compute Unit) 160 GB instance storage 32-bit or 64-bit platform Extra Large $0.720 per Hour 15 GB memory 8 EC2 Compute Units (4 virtual cores with 2 EC2 Compute Units each) 1,690 GB instance storage 64-bit platform

There are also additional costs for I/O and extra storage (although these aren't much). You can start/stop the machines and should generally only pay when in use. Cloud computing Very low entry cost, though you dont own the machines. Flexible resource levels. Someone else maintains and secures the machines. Usually not connected directly to useful resources. You dont know what they are doing with your data, and usually they are hosted

outside your country, which may cause data-protection issues. Latency between machines can vary, though it is often possible to request machines local to each other. Issues with architecture Is there going to be a lot of communication? LAN Beowulf (or bus-connected supercomputer). Can you cope with security issues? If not, Grid or Cloud. What skills do you need? If not Linux, then Beowulf-lite MPJ on a Windows cluster.

Do you have the computing resources? If not, Volunteer system, Grid or Cloud. What other services do you want? If many, probably Grid. Do you want a permanent resource? If not, Volunteer, Grid, or Cloud. Further info A really good book that introduces threads, concurrent.futures, and celery, and gives a detailed look at setting up Amazon cloud computing is:

Francesco Pierfederici (2016) Distributed Computing with Python. It doesn't cover MPI. For this, the MPI4Py docs are good. You could also check out a general MPI book, as the API and issues are roughly similar, and you'll need to get a C version running in the background. A good one is: Peter Pacheco (2011) An Introduction to Parallel Programming (update on Parallel Programming with MPI? C++ code, but fine).

Recently Viewed Presentations

  • The Vertebrate Animals

    The Vertebrate Animals

    More than 120 species have likely gone extinct since 1980. Class Amphibia FROG DISSECTION CLASS REPTILA Reptiles 6500 species, appeared 300 mya Cold-blooded 3 chambered heart Body covered with scales.
  • Limb Preservation

    Limb Preservation

    The usefulness of OBL is now established. To improve care and convince the payer of quality care being rendered through outpatient centers a comprehensive approach is required in management of limb ischemia. We have to move from the concept of...
  • Crosby/Krieger - Idaho State University

    Crosby/Krieger - Idaho State University

    Mapping of hazardous rockfall areas has been completed in a few areas around the world. Rock-bounce calculations and estimation methods for delineating the perimeter of rockall zones have also been determined and the information widely published.
  • High Precision Radiometer Space Instruments, Inc., Encinitas, Ca

    High Precision Radiometer Space Instruments, Inc., Encinitas, Ca

    Non-Volatile, Solid-State Recorder for Spacecraft Seakr Engineering, Inc. Englewood, CO ACCOMPLISHMENTS Developed software to control flash memory in a low earth environment. Designed flash based memory for operation in a low earth orbit. Delivered flash based solid state recorder for...
  • Présentation PowerPoint

    Présentation PowerPoint

    Paule-Françoise de Gondi et Olympe de Mancini à St Cloud. Jean-François Paul de Gondi est l'oncle de Paule-Françoise de Gondi, duchesse de Lesdiguière. Ils vont se rendre ensemble ou séparément au château de . St-Cloud mais jamais en silence. Le...
  • West Coast University BSN Program NURS 120

    West Coast University BSN Program NURS 120

    External Hemorrhoids . Located underneath the skin that surrounds the anus. Can be felt when they swell and may cause itching or pain with a bowel movement, as well as bleeding. A thrombosed external hemorrhoid occurs when blood within the...
  • Blanket Purchase Agreement - Usda

    Blanket Purchase Agreement - Usda

    The Mobitask App. Very brief overview of what we will talk about in more depth later in the Webinar. This is the MobiTask app . It is a free app that works on any platform. We mainly utilize the e-Forms...
  • Chapter 10 Recipes - Blueprints for food

    Chapter 10 Recipes - Blueprints for food

    A recipe is a list of ingredients and directions for preparing food. The 5 parts of a recipe are the ingredient list, cooking equipment needed, cooking time and temperature, steps to follow, and yield. Recipe terms tell you exactly how...