As a goal, we would like to guide realization of an energy system that is interoperable, by standardizing the meaning of data exchanged and guide implementation of open source projects developing exchange (interfaces) based on these standards. We focus on the business side of interoperability as common understanding, trust and transparency of data used and data exchanged is a fundamental part of the energy system of the future, regardless of geography. |
Teleconference weekly on Thursdays, 7:00 US Pacific Time. Meeting details can be found in the LF Energy Community Calendar or below.
________________________________________________________________________________
Join Zoom Meeting: https://zoom.us/j/91818399821?pwd=TVBEeFdLU0IzdlE5cTZjTVIvM0Z6dz09
Meeting ID: 918 1839 9821
________________________________________________________________________________
If you like to add you topic to add to the agenda, feel free to add it.
We will create a parking lot, with topics of interest and then vote on priority.
recent OEO paper: Booshehri, Meisam, et al (1 September 2021). "Introducing the Open Energy Ontology: enhancing data interpretation and interfacing in energy systems analysis". Energy and AI. 5: 100074. ISSN 2666-5468. doi:10.1016/j.egyai.2021.100074. Open access.
Presentation of Ibrahim Haddad (Executive Director, LF AI & Data Foundation) - presentation on LF AI and Data
Session to prepare questions toward second OSDU session.
OSDU LFEnergy Architecture Introduction v2.pptx
Questionnaire
Questions LFE | Answers Shell (dd 5 February 2021) |
---|---|
GENERAL | |
What components are open source frameworks based and what are Schlumberger Delfi developed or Shell developed? What does Delfi / Shell developed mean code wise? | Shell developed means an API specification and implementation of a data platform developed with Shell called SDU. It was started in early 2018 and contributed to the OSDU Forum in early 2019 as the initial baseline for OSDU development. DELFI developed means it was originally developed in Schlumberger to support the DELFI environment to support both internal and commercial upstream workflows. Internally, the code was referred to as the Data EcoSystem (DES) and has been in development since 2015 and contributed to the OSDU Forum in late 2019. Because the SLB code base had both a larger footprint and contained an implementation of business logic deeper than the API specification, it was chosen to become the basis for future OSDU development. The components in R3 that were contributed from Schlumberger to this space include what is referred to as
Components from Shell’s implementation that were not present in the Schlumberger implementation that are being ported/redeveloped on top of the new code, include the Delivery Service and support for Global (multi-region) deployment. Most of the other Shell contributed services had a corresponding SLB implementation and thus the deltas were treated as requirements. In addition to Shell and Schlumberger, there have been existing code contributions from other companies as well. Most notably, this included an optimized cloud friendly bulk seismic representation (OpenVDS) from Bluware. Beyond the initial code contributions, there are other development activities that started in Open Source, including:
|
What function has OFP in total architecture? |
|
How do you see OSDU be adopted and used in LFE community? I mean focus with current community is with oil and not with power right. Wat is goal of Shell to put it forward at LFE? | - |
How is lineage implemented? How is discovery implemented? How is meta date capturing and usage implemented? | Re: Lineage First, there is a notion of versioning which is distinct from lineage.
Version is managed by the system’s support for immutability. Lineage is largely a usage convention agreed by the community of developers and data modelers. It is captured as references to ancestor records in the metadata. There is a lot of interest in supplementing this with an activity model which can be more expressive than simply linkage. Re: Discovery All metadata structures in the system are described in the Schema service. When a metadata record is stored (storage service), it triggers an event. An indexing service listens to the event, parses the metadata record according to the structure defined in the Schema service and maps it to an ElasticSearch index which can then be used for search. This makes the system flexible in that a user defines a structure, and this structure is then used directly for search without the need for custom indexers. If custom indexers are required for unique search patterns (such as full geospatial (GIS) rather than elastic spatial, then they can be accommodated by creating new consumption zones (a different topic). |
Security standards for all data, in Rest, transit in use? Any cryptograph standard implemented and evolution on roadmap? | You can see the Open Source version of the security requirements at https://community.opengroup.org/groups/osdu/platform/security-and-compliance/-/issues?scope=all&utf8=%E2%9C%93&state=opened&author_username=pacohope There is a security standard document on the member side (you must be a OSDU Member to read it while it is in development) that is a formal version of the requires as well as the cloud providers response on how they meet the requirements. Security (and operations) has a shared responsibility model; so, some aspects can be addressed in the Open Source code; others are addressed at time of deployment and yet others during operations. Given that this is an Open Source development, compliance is enforced through certification of a deployable system. A deployed system that does not meet the security standards defined by the OSDU Forum cannot be certified. |
Future roadmap. How does development of platform look like including view on backwards compatibility | The initial developments were primary focused on a backlog of contributions of existing code. These contributions will continue as the backlog of contributions in the Upstream Oil and Gas space has not been exhausted (far more to go). The members of the Forum itself are also pivoting towards becoming carbon neutral and embracing new energy sources; so we can expect to see contributions and new developments in this space. The system supports this through a variety of means.
|
Could you elaborate on the D-WIS | - |
What is the programming language is used to transform datasets? | The system is microservices, so accommodative of many languages. However, development convention has primarily settled on 3 languages.
|
DATA PRINCIPLES | |
What does minimum viable governance mean? Does it exclude type of use cases because if it (compliancy for instance) or does it adept to what is needed? | Minimum – means whatever is necessary with a bias towards making it easy to bring in new data. Concretely, it means describing your data making it discoverable and tagging the data to understand whether you have the right to use it. Policies can be additive, since you always have the right to reject data coming in the system, or add additional policies on usage later (new entitlements engine). The principle was introduced because of the tendency in the past to refuse the admission of anything less than perfectly described, perfectly formed data. |
Is focus on Interoperability not a business and IT goal to focus, or what principles are in place for interoperability purpose? | Interoperability is not a principle, because it is the primary goal of the system.
Principles help us decide on competing / alternative approaches towards achieving the goal |
Why is data mesh / user (group) centricity not reflected in data principles? | Maybe because the principles were written 5+ years ago, and the notion of data mesh as an implementation pattern came in later. Data-mesh is an architectural/design decision. |
How are generic types supported (details). | The data structures from the records are declarative and registered via the schema service with corresponding metadata records stored in the storage service which can then be discovered via the search service . The goal is to be able to represent anything described in well-formed JSON; although there are complications when trying to map JSON to ElasticSearch indices to consider. Some tutorials on how this works can be found in |
DATA ARCHITECTURE BASICS | |
If used with LFE, the platform is a separated one from the oil focus instance? For what purpose are the concepts kinds and namespaces used in the platform? | there is no notion of the single deployable “instance” in OSDU. Different companies will find different ways of deploying it, whether it be multiple regions for within a company, multiple companies in a SaaS or national repositories. There is, currently, one software baseline officially supported in Open Source; and given the rapid rate of development of the core platform, it is premature to think about forking and developing in parallel. What can be done (by design) is independently developing data representations, ingestions, enrichments, optimized domain services that plugin to the core platform. These can be developed anywhere – and probably should be developed close to the community of interest. Re: Namespaces The purpose of “kinds” is to support the addition of new data representations to the platform as a user function rather than a development one. This allows for the footprint to expand based on the scope of interest of the consumer. The naming convention for a kind (namespace, authority, …) is intended to provide context into the type of data that the kind is describing. For instance, for electricity, you could think about capturing the packages and entities defined in CIM as a set of kinds. |
Are there semantic standards in use of the Oil & Gas industry, if so how are they used in the platform? | AD |
What meta model is used to describe the semantics? What tools are used describe the semantics? | AD |
What meta model is used for the provenance/lineage? W3c PROV? | AD |
How are API's generated out of the metamodel? | There are two types of APIs,
The contract that binds these together is that the actual persistence of data in the platform from the domain specific APIs complies with the requirements to be discoverable, described…. |
What semantics are used to publish the discovery services? DCAT? | The semantics are described in the definition and registration of JSON metadata models in the Schema services and the mapping is performed by model specific services (such as mapping to elastic search). |
“Well Known Schemas (WKS)”; Can you give example? Do you have a list? | A WKS is a community defined model.
|
CONCEPTS / PATTERNS | |
The concept CQRS is used at application level implementation, right? The generic storage layer / data distributed layer is decoupled from this | The concept is deeper than just the typical view of an application layer. The ability to expose multiple metadata records on the same data with different views represents a platform level of CQRS. The simplest example would be search, where the properties used in search can be narrower, or different from the properties initially loaded into storage. Keep in mind that “applications” can also run inside the platform producing (Command) new representations for specific consumers (Query). |
DATA OWNERHIP | |
Who owns the data? | the system can be deployed by anyone. Whoever runs the instance is liable for the data (note: the notion of data ownership vs. data liability are different since the platform is often used to host licensed data by users and thus the owner is an external contractual; entity. |
Is the platform multi-tenant? | yes, some providers of the platform are SaaS providers and thus need to host multiple customers in a single instance for cost/operations reasons (there is support for string isolation between tenants). Even in a single customer deployment, different partitions can be used to separate business units, contractual data, business continuity needs, …. |
Does the platform control somehow personal data (privacy affected)? | the system has no direct recognition of or support for personal data (PII). In Upstream technical computing, personal data is peripheral to workflows of interest. By convention, any personal data loaded into the system should remain incidental to the purpose of the platform unless stronger controls are added. Personal information does come into strong play with respect for security and auditability. Such logs that contain this information should be protected. |
REAL TIME DATA | |
Real-time and streaming services D-WIS* (details/roadmap) | - |
OPEN DATA LABEL (from Robbie Morrison) | On the "open data" label: My questions to Shell related to the legitimacy of using the phrase "open data" when the data held — aside from that mandated public domain by certain national authorities — was governed by brokered contracts restricting its use and further publication. Indeed a central design feature of the OSDU was the tagging of data with its contractual attributes to automate this process — my reading admittedly. There are clear touchstone definitions for "open data", one community‑driven from the Open Knowledge Foundation (OKI) and another stated in recital 16 of the Open Data Directive 2019/1024. This question was discussed in an engaged manner but I did not record a clear answer (although I am happy to be corrected). The term "open data" is not protected by legislation, so instead I suggested perusing a voluntary "code of practice for open data in the energy sector" under Directive 2005/29/EC on Unfair Commercial Practices. The feasibility of this option is still be be determined. On data ownership more generally: Worth noting too that data cannot be "owned" as such (some questions were also related to data ownership). Setting aside European database rights for the moment, data is either private, its movement restricted by contract (as per the OSDU proposal), subject to human rights considerations when personal (under the GDPR), or publicly available. Certain data needs to be published under statutory reporting for public interest reasons (usually to address market failure) which is why some electricity sector data needs to be provided to the ENTSO‑E Transparency Platform. Notwithstanding, other legal considerations, such as unfair completion, various civil wrongs, and misappropriation may apply, but these are not fundamentally matters underpinned by data ownership. |
Topics for the meeting of 28 January 2021
Welcome
Agenda:
Presentation of Johan Krebben, IT CTO, Royal Dutch Shell on OSDU (Open Surface Data Universe) presentation. To be decided how much time needed.
Paper on Equinor (was Stadoil and StadoilHydro) relational database Slegge containing subsurface data:
My blog on SzenarienDB which uses the same ontology-based data access:
Welcome
Set the agenda
Welcome
Set the agenda
storyline TAC document
Welcome
Set agenda
Plan timeslot in TAC on proposal on data architecture guidance on 2nd of February
Johan Krebben, IT CTO, Royal Dutch Shell. OSDU (Open Surface Data Universe) presentation.
Status of document proposal to present to TAC on data architecture guidance
Robbie Morrison was asked to post details of the e-Gridmap video he had briefly reported on:
The YouTube URL is marked at my 2 minute long question on open sourcing the underlying code. Elering could well be interested in doing so.
Welcome
Set the agenda
Leadership: who likes to take over the role of Sander
TAC presentations: who does what
Use approach of LFEdge? https://wiki.lfedge.org/display/LE/Project+Stages%3A+Definitions+and+Expectations
Use set of standards and maturity model : Update on the maturity model document uploaded)
Use phases of LFEnergy.
From phase 3 comply to standards on various levels you can standardise on
Very short reports from Robbie Morrison on:
Welcome
Set the agenda
Jonas van den Bogaard will join us on how we can bring the data architecture to the TAC
Very short reports from Robbie Morrison on:
Rob's improvement
Notes:
Jonas: 30-60 minutes introduction into data architecture. Jonas can put us on the agenda.
Perspective: project use focused (how can projects use/benefit from the data architecture; why); some slides would be nice.
Discussions on the next steps could be discussed later.
Landing page links for the above short reports from Robbie Morrison:
Robbie Morrison also mentioned the frictionless data concept from the Open Knowledge Foundation (OKFN):
Rob presented an idea for an maturity framework for LF energy data architecture aspects.
The maturity model
Naveen Goswamy suggested to look into grid‑wise architecture model and map these models
We need to be careful to ..
Exchange methods need more clarity, particularly in relation to control signalling.
Rob will share his proposal to fill in.
Welcome
Set the agenda
Notes:
Rob's suggestions in terms of interoperability maturity:
1 datamodel
2 description / definition
3 exchange language (Technical format, e.g. JSON)
4 exchange method (E.g. filesharing / API)
Open standards are only allowed. Proprietary standards are not allowed.
Eric made his first changes to the document to make it ready to discuss it with Jonas (TAC).
Welcome
Set the agenda
Notes:
The current architecture guideline only provide direction. More can be done (e.g. testing).
Maturity levels suggestions:
1 collaboration with other projects within the ecosystem (e.g. LF energy project or other related projects)
2 Using standards (if exists)
3 Test the interoperability between projects
Projects can go to the data office hours to get help to the next phase.
Actionlist:
To do | Who |
---|---|
Review the document for the TAC | Who did not do it so far |
Ask the TAC chair to join the office hours to discuss the TAC recommendations | Sander |
Write a TAC Recommendation list | Sander & Rob (after 3 December) |
How can we bring the architecture document to the TAC?
What form/ how? When?
Do we need to split it up?
Welcome
Set the agenda
Discuss the next steps how to move the data architecture to the next level
Actionlist:
To do | Who |
---|---|
Review the document for the TAC | All |
Write a TAC Recommondation list | Sander & Rob |
Aks the TAC:
What is to way to inprove the interobability of the LF energy projects?
Current ideas:
Interesting project to look at:
Welcome
Set the agenda
What is our propose? (discussion last week)
Do we want to add SARGON to the referance models?
Notes:
Discussion how we bring the data architectur to a new level.
Not so much operating modes,but more possible practical guidelines, methods:
<<Please add your suggestion to the list; How can we bring the data architecture to a new level within LF Energy and related projects>>
Welcome
Set the agenda
Presentation of the SARGON onlology
What is our propose? (discussion last week)
Notes:
Dainiel is looking into this project. It might be relevant for our community.
https://www.data-infrastructure.eu/GAIAX/Navigation/EN/Home/home.html
Slides of the SARGON ontology:
Here are the likes into the projects, repos, and mentioned publication respectively.
Welcome
Set the agenda
Presentation of the SARGON onlology
Outline of deliverables + time horizon
Welcome
Set the agenda: are their any topics for today?
Status of the invite of: Operator fabric / OpenEEmeter / Powsybl
Unfortunately Robbie Morrison cannot make this meeting due to a clash. But a couple of comments nonetheless:
Notes: Gianluca will each out to check on the SARGON onlology
Sander will contact the invites of other projects.
Welcome
Set the agenda: are their any topics for today?
Status of the invite of: Operator fabric / OpenEEmeter / Powsybl
Discussion: What needs to be added to the document to bring it to the TAC?
Open source and Standards update CEN / CENELEC
Marc Costa
Notes:
Gianluca will ask somebody form SARGON to give some explanation in this group.
Derrick made a docker container for CIMtool.
The Relationship Between Open Source Software and Standard Setting
The upcomming meetings are cancelled due to the availibility of Sander
Daniel Lázaro can be reached by the invited projects if they want to present their projects and talk about semantics in the cancelled period.
Notes:
We decided to share the relevant slides this wiki. A decision on how to record video is not needed.
CIMTool development:
New agenda items can be added to this group.
Notes:
Connecting to the project is use-full, However the OEDI and RIAPS had little relation with the data architecture.
OpenEEmeter, PowSyBl and OperatorFabric will be invited to present their project as well. Sander Jansenwill invite them.
Opentaps project: https://opentaps.org/2020/01/30/green-button-xml-openeemeter-added-opentaps-mv/
Derrick Oswald will reach out to the maintainers of CIMtool to ask them to join a office-hour to talk about their roadmap and collaboration.
Proposed agenda : one hour later at 08:00 PDT -0700 and 17:00 CEST +0200:
Michael Rossel of OEDI
Summary of the OEDI project:
Gabor Karsai form the RIAPS project
Summary of the RIAPS project:
Slides:
Proposed:
Meeting notes:
Review the M490 SGAM introduction to the document Eric Houmeswill check this later on
Sander Jansen and Eric Houmes will start with the TAC. Based on the TAC converstations, we can change this
The COM-PAS project was presented. The COM-PAS slides can be found here.
Robbie Morrison made some comments about open data (see below)
Robbie Morrison will make suggestions to record the workgroup meetings form a legal an practical pespective.
Sander Jansen will request @Michael Rossel to join the 3th of september and asks the RIAPS project to join next week
Daniel Lázaro will contact RAIPS for setting the expectations to join next week.
Follow up:
(I, Robbie Morrison, was asked to record my comments about developing country transfers, please move this content elsewhere if this is not the best place. Or alternatively delete this comment.)
There are three projects promoting open source/data/access for energy policy development for developing country:
References
IEEE (6 July 2020). Global Power Systems Transformation Consortium: landscape analysis of relevant standards, testing, and certification activities. IEEE. Closed access.
OPM and UK Aid (August 2019). Synthesis Report on the "Fourth Roundtable Discussion on Strategic Energy Planning" (Trieste 28 June 2019). United Kingdom: Oxford Policy Management and UK Aid. U4RIA Section 4 (pages 7–8) titled "Working groups on modelling standards and capacity building".
Proposed:
Meeting notes:
Everybody agrees with the Working group goal with the addition of adding open source.
Daniel Lázaro will ask the RIAPS project to join the office hour to talk semantics (see the OEDI initiative).
Sander Jansen will add a basic introduction to the SGAM in the data architecture document.: Gianluca Dianese and Naveen Goswamy can you guys review and improve it.
Eric Houmes will connect with the LF energy guys to see if we can check against standards/information models
More information on LF energy project activity can be found here: https://lfanalytics.io/projects, Other examples can be found here.
Topics:
Data Architecture Working Document
Data Architecture Community Folder
Name | Company |
---|---|
Sander Jansen | Alliander |
TenneT | |
Daniel Lázaro | OSIsoft |
Alliander | |
Alberto Abella | FIWARE |