#datazones

The trust of businesses in the Azure OpenAI Service was increased by the implementation of Data Zones.

Data security and privacy are critical for businesses in today’s quickly changing digital environment. Microsoft Azure OpenAI Service provides strong enterprise controls that adhere to the strictest security and regulatory requirements, as more and more businesses use AI to spur innovation. Anchored on the core of Azure, Azure OpenAI may be integrated with the technologies in your company to assist make sure you have the proper controls in place. Because of this, clients using Azure OpenAI for their generative AI applications include KPMG, Heineken, Unity, PWC, and more.

With over 60,000 customers using Azure OpenAI to build and scale their businesses, it is thrilled to provide additional features that will further improve data privacy and security capabilities.

Introducing Azure Data Zones for OpenAI

Data residency with control over data processing and storage across its current 28 distinct locations was made possible by Azure OpenAI from Day 0. The United States and the European Union now have Azure OpenAI Data Zones available. Historically, variations in model-region availability have complicated management and slowed growth by requiring users to manage numerous resources and route traffic between them. Customers will have better access to models and higher throughput thanks to this feature, which streamlines the management of generative AI applications by providing the flexibility of regional data processing while preserving data residency within certain geographic bounds.

Azure is used by businesses for data residency and privacy

Azure OpenAI’s data processing and storage options are already strong, and this is strengthened with the addition of the Data Zones capability. Customers using Azure OpenAI can choose between regional, data zone, and global deployment options. Customers are able to increase throughput, access models, and streamline management as a result. Data is kept at rest in the Azure region that you have selected for your resource with all deployment choices.

Global deployments: With access to all new models (including the O1 series) at the lowest cost and highest throughputs, this option is available in more than 25 regions. The global backbone of the Azure resource guarantees optimal response times, and data is stored at rest within the customer-selected

Data Zones: Introducing Data Zones, which offer cross-region load balancing flexibility within the customer-selected geographic boundaries, to clients who require enhanced data processing assurances while gaining access to the newest models. All Azure OpenAI regions in the US are included in the US Data Zone. All Azure OpenAI regions that are situated inside EU member states are included in the European Union Data Zone. The upcoming month will see the availability of the new Azure Data Zones deployment type.

Regional deployments: These guarantee processing and storage take place inside the resource’s geographic boundaries, providing the highest degree of data control. When considering Global and Data Zone deployments, this option provides the least amount of model availability.

Extending generative AI apps securely using your data

Azure OpenAI allows you to extend your solution with your current data storage and search capabilities by integrating with hundreds of Azure and Microsoft services with ease. Azure AI Search and Microsoft Fabric are the two most popular extensions.

For both classic and generative AI applications, Azure AI search offers safe information retrieval at scale across customer-owned content. This keeps Azure’s scale, security, and management while enabling document search and data exploration to feed query results into prompts and ground generative AI applications on your data.

Access to an organization’s whole multi-cloud data estate is made possible by Microsoft Fabric’s unified data lake, OneLake, which is arranged in an easy-to-use manner. Maintaining corporate data governance and compliance controls while streamlining the integration of data to power your generative AI application is made easier by consolidating all company data into a single data lake.

Azure is used by businesses to ensure compliance, safety, and security

Content Security by Default

Prompts and completions are screened by a group of classification models to identify and block hazardous content, and Azure OpenAI is automatically linked with Azure AI Content Safety at no extra cost. The greatest selection of content safety options is offered by Azure, which also has the new prompt shield and groundedness detection features. Clients with more stringent needs can change these parameters, such as harm severity or enabling asynchronous modes to reduce delay.

Entra ID provides secure access using Managed Identity

In order to provide zero-trust access restrictions, stop identity theft, and manage resource access, Microsoft advises protecting your Azure OpenAI resources using the Microsoft Entra ID. Through the application of least-privilege concepts, businesses can guarantee strict security guidelines. Furthermore strengthening security throughout the system, Entra ID does away with the requirement for hard-coded credentials.

Furthermore, Managed Identity accurately controls resource rights through a smooth integration with Azure role-based access control (RBAC).

Customer-managed key encryption for improved data security

By default, the information that Azure OpenAI stores in your subscription is encrypted with a key that is managed by Microsoft. Customers can use their own Customer-Managed Keys to encrypt data saved on Microsoft-managed resources, such as Azure Cosmos DB, Azure AI Search, or your Azure Storage account, using Azure OpenAI, further strengthening the security of your application.

Private networking offers more security

Use Azure virtual networks and Azure Private Link to secure your AI apps by separating them from the public internet. With this configuration, secure connections to on-premises resources via ExpressRoute, VPN tunnels, and peer virtual networks are made possible while ensuring that traffic between services stays inside Microsoft’s backbone network.

The AI Studio’s private networking capability was also released last week, allowing users to utilize its Studio UI’s powerful “add your data” functionality without having to send data over a public network.

Dedication to Adherence

It is dedicated to helping its clients in all regulated areas, such as government, finance, and healthcare, meet their compliance needs. Azure OpenAI satisfies numerous industry certifications and standards, including as FedRAMP, SOC 2, and HIPAA, guaranteeing that businesses in a variety of sectors can rely on their AI solutions to stay compliant and safe.

Businesses rely on Azure’s dependability at the production level

GitHub Copilot, Microsoft 365 Copilot, Microsoft Security Copilot, and many other of the biggest generative AI applications in the world today rely on the Azure OpenAI service. Customers and its own product teams select Azure OpenAI because it provide an industry-best 99.9% reliability SLA on both Provisioned Managed and Paygo Standard services. It is improving that further by introducing a new latency SLA.

Announcing Provisioned-Managed Latency SLAs as New Features

Ensuring that customers may scale up with their product expansion without sacrificing latency is crucial to maintaining the caliber of the customer experience. It already provide the largest scale with the lowest latency with its Provisioned-Managed (PTUM) deployment option. With PTUM, it is happy to introduce explicit latency service level agreements (SLAs) that guarantee performance at scale. In the upcoming month, these SLAs will go into effect. Save this product newsfeed to receive future updates and improvements.

Post: 14 January 2013

Data licensing can be tricky.

There are data sets that are clearly open data and data sets that are clearly not. 

Then there are those that have potential, if you look at them in the proper light ...

The Indices of Multiple Deprivation (IMD) are a multi-domain measure of relative levels of deprivation within small geographic areas of the UK.

Indices for England, Scotland, Wales and Northern Ireland are produced separately by Government and have slightly difficult criteria, but it's possible to make comparisons across the data for different countries.

IMD data sets are used widely in the public sector and for academic research. I've used IMD data myself as inputs into risk models for the insurance industry. The crime domain in particular is useful in predicting geographic variability of economic losses to theft, civil disorder, etc.

Last month the 2012 version of the Scottish IMD was released by the Scottish Government. Alasdair Rae has put together a useful website that visualises the SIMD 2012 data interactively on a map.

In England and Wales the small-area geography used in the IMD is the Lower Super Output Area. LSOA boundaries are unambiguously available as open data, and may be downloaded from the ONS website and elsewhere.

However in Scotland the unit of geography underlying the IMD is called a Data Zone. The Data Zone boundaries currently in use were produced in 2004 but built up from 2001 Census output areas.

There's background information on Data Zones here and here. This is what they look like:

Pretty much all administrative geography for England, Scotland and Wales is now available as open data. So I was surprised to see the restrictive licensing terms presented on the download page for the Scottish Government's Geography Data (which includes the Data Zone boundaries).

You can read the licence on the Scottish Neighbourhood Statistics website. (Click on Go to Data Download, then Download Geography. The SNS site doesn't like direct links.)

The Geography Data licence starts like this:

We the Scottish Government grant you a non-exclusive non-transferable licence (without the right to sublicense) to copy and use the Data which is derived from Ordnance Survey data and as such is subject to the terms and conditions of the licence agreement between The Scottish Government and Ordnance Survey.

No transferability or sub-licensing, so right away that says it's not an open licence. Without transferability it would be very difficult to put the Geography Data on a public website in a legally compliant manner.

The wording tells us clearly that the download includes Ordnance Survey derived data. However it's ambiguous whether the Scottish Government is claiming any additional database rights.

The licence then says:

For any use of this material a Click-Use PSI Licence is also required.

with details of how to arrange that via the Office of Public Sector Information.

The problem is that the Click-Use Licence system was phased out from 2010, when the Open Government Licence (OGL) was introduced. The link provided on the SNS's download page now resolves to a National Archives page about the OGL.

Click-Use Licences were typically issued for a five-year period, so in theory there could be re-users out there using the SNS Geography Data under those terms. However the OPSI is no longer issuing new Click-Use Licences.

Interpreted literally that means nobody who downloads the Geography Data subsequently can comply with the licensing terms on the download page.

It's fairly obvious the licence on the download page is out of date. Does that mean we can substitute the terms of the Open Government Licence? That would enable us to do away with the restrictions on transferability and sub-licensing, both of which are supported by the OGL.

There is certainly a convention that the OGL simply replaces the Click-Use Licence, and we have statements to that effect from National Archives.

But I've never been entirely comfortable that the Government nailed that down properly, and there are substantial differences between the Click-Use Licence and the OGL. The Click-Use Licence does not specifically set out the scope of permitted re-use, and it seems to exclude mapping data. National Archives has also muddied the waters by introducing additional licences to the UK Government Licensing Framework.

Back to the Geography Data licence:

You may only use the Data for your own internal business use, that is use of Data for the internal administration and operation of your business and not for any commercial purpose, and not for financial profit or gain. Financial gain would include any profit whether direct or indirect, or benefit from the use or publication of the Data in any form.

This tells us that when the licence was written there was no intention to automatically allow full commercial re-use of the Geography Data, as would be permitted under the OGL.

So this addendum:

For any other use of the Data you will need to obtain permission from Ordnance Survey

The remainder of the licence is boilerplate to emphasise key terms from the Click-Use Licence.

Doesn't look much like open data, does it?

This is why we shouldn't always take data licences at face value. Licences are contracts, and subject to argument and interpretation.

In this case a literal reading of the licence would prevent anyone from actually re-using the data even for internal business purposes, because the Click-Use Licence is no longer available. It's a reasonable assumption this is not the intention either of the Scottish Government or the Ordnance Survey. So there's something wrong with the licence.

This is where it helps to understand the historical context, and the data specifications themselves. Reading the background information gives us some idea of which Ordnance Survey data was used to produce the Data Zones and, although it's not conclusive, it looks as if it was simply older versions of data that is now included in Ordnance Survey's open Boundary-Line product.

I did the sensible thing and sought an opinion from the Ordnance Survey. I had to persist a little but in due course received a detailed and helpful e-mail.

The key points from the Ordnance Survey e-mail are as follows:

Although we are unable to confirm which Ordnance Survey data has been used to create this data, we are happy to treat this data as being part of OS Boundary-Line and covered by the Open Data Licence.

As there is additional data present, we would always recommend that you seek permission of the map's owner (in this case SNS) to ensure that there is no additional copyright enforced by them. It would appear that, as the link provided within SNS' download licence (which refers to "Click-Use PSI Licence) is now for the Open Government Licence, the data is covered by the Open Government Licence ...

On the strength of the above I would personally be confident in treating the SNS Geography Data as open data. I think Ordnance Survey permission to apply the OS OpenData Licence was the only real barrier.

It is significant that the Ordnance Survey is willing to apply the OS OpenData Licence to older versions of Boundary-Line, and presumably also to older versions of the other OS OpenData products. That makes practical sense but I don't think it is stated explicitly on the OS website.

As regards "additional data present", it would certainly be better to have a re-written licence on the download page itself to remove any remaining ambiguity. However based on the existing wording, and with due regard to the remarks in the OS's e-mail, I think there is a perfectly defensible argument that the Scottish Government has left the full licensing decision to Ordnance Survey.

_______________________________

Update 11 July 2013:

The Scottish Government has now updated the Data Zones 2001 metadata record on Data.gov.uk to confirm that the dataset is re-usable under the OS OpenData licence.