Feature #21617: Add Internationalized Domain Name (IDN) support to URI

• Status: Open
• Assignee: -
• Target version: -

[byroot (Jean Boussier)]

Originally proposed by @chucke at https://github.com/ruby/uri/issues/76, trying to formalize it here.

Context

Internalized Domain Names, are getting more common, yet Ruby's uri default gem has no support for it:

>> URI("https://日本語.jp/")
URI must be ascii only "https://\u65E5\u672C\u8A9E.jp/" (URI::InvalidURIError)

So any program that which to handle arbitrary valid URIs provided by users can't use the uri gem, and instead have to depend on third party gems like addressable

>> Addressable::URI.parse("https://日本語.jp/")
=> #<Addressable::URI:0xd648 URI:https://日本語.jp/>

But even there, it won't seamlessly work with other libraries such as net-http:

``ruby

Net::HTTP.get(Addressable::URI.parse("https://日本語.jp/")).bytesize
OpenSSL::SSL::SSLSocket#connect_nonblock': SSL_connect returned=1 errno=0 peeraddr=[2001:218:3001:7::110]:443 state=error: ssl/tls alert handshake failure (SSL alert number 40) (OpenSSL::SSL::SSLError)

You have to explicitly normalize the URL:

```ruby
>> Addressable::URI.parse("https://日本語.jp/").normalize
=> #<Addressable::URI:0x130d0 URI:https://xn--wgv71a119e.jp/>
>> Net::HTTP.get(Addressable::URI.parse("https://日本語.jp/").normalize).bytesize
=> 8703

Feature Request

I believe it's would be very useful if the default uri gem had the capacity of:

• Parsing IDNA domain names.
• Convert URLs between their unicode and ASCII forms.

The URI::Generic class already have a #normalize method to ensure the host and schema parts are all lower case, it could be extended to encode IDN hosts into their ASCII equivalent.

It would also be useful if the opposite operation was supported for display purposes, not sure what name such a method could have, perhaps canonicalize?

Implementation

In https://github.com/ruby/uri/issues/76 @skryukov pointed to his pure Ruby implementation of IDNA 2008 (https://github.com/skryukov/uri-idna), I believe it would be good to upstream parts of it in the uri gem to implement these feature.

[byroot (Jean Boussier)]

• Description updated (diff)

[byroot (Jean Boussier)]

@skryukov also pointed to me the existence of https://github.com/y-yagi/uri-whatwg_parser by @y-yagi

[chucke (Tiago Cardoso)]

Just adding my original public API suggestions, for visibility and further discussion by the core team.

I propose that URI::Generic supports punycode decoding OOTB by relying on the current behaviour of URI::Generic#hostname, which already applies transformations to the passed host when necessary, such as in the below case of IPv6 addresses:

# the example above is inspired in how uri already handles IPv6 addresses
uri = URI("https://[::1]")
uri.host #=> "[::1]", cannot be used in Socket.new(host, port)
uri.hostname #=> "::1", can be used in Socket.new(host, port)

therefore, punycode translation would happen transparently for IDNAs when calling hostname:

uri = "https://l♥️h.ws"
uri = URI(uri)
uri.host #=> "l♥️h.ws" #=> cannot be used in Socket.new(host, port)
uri.hostname #=> "xn--lh-t0xz926h.ws" #=> can be used in Socket.new(host, port), which will perform DNS via getaddrinfo

This would require very little change in resolv library, before issuing the DNS query. The same would apply for most use cases, I believe.

The required punycode decoding logic could be implemented in a separate URI::Punycode module. This module could be exposed publicly, with a single public method, decode(uri), which would return the punycode URI of a given IDNA. This API could be extended to support more advanced use cases beyond the main common use case (which URI::Generic#hostname should address), like the ones documented here.