WO2016057760A1

WO2016057760A1 - Reliable and efficient motionless image transfer for sip/rtp media conferencing

Info

Publication number: WO2016057760A1
Application number: PCT/US2015/054639
Authority: WO
Inventors: Joseph A. OLSON; Michael E. KIM
Original assignee: Infinesse Corporation
Priority date: 2014-10-08
Filing date: 2015-10-08
Publication date: 2016-04-14

Abstract

The present invention when integrated into a distributed SIP/RTP Conferencing System provides an efficient and reliable transport for motionless images as well as significantly improve the image quality of single image data frames. The invention is well suited for high quality still video images captured from either a digital camera device or a bit map image of all or a portion of a video graphic display memory. The invention provides several key improvements to standard RTP video media streaming techniques in terms of network efficiency, computer processing loading, reliable image delivery, image quality, participant privacy, and distractions from unnecessary remote participant motions. The invention provides packet image caching in conference routing server to provide efficient image updates to participants when joining an established conference and also to resend images to targeted participants with network transmission failures. All motionless image media formats are supported.

Description

RELIABLE AND EFFICIENT MOTIONLESS IMAGE TRANSFER FOR SIP RTP

MEDIA CONFERENCING

NOTICE OF COPYRIGHTS AND TRADE DRESS

[0001] A portion of the disclosure of this patent document contains material which is subject to copyright protection. This patent document may show and/or describe matter which is or may become trade dress of the owner. The copyright and trade dress owner has no objection to the facsimile reproduction by anyone of the patent disclosure as it appears in the Patent and Trademark Office patent files or records, but otherwise reserves all copyright and trade dress rights whatsoever.

RELATED APPLICATION INFORMATION

[0002] This patent claims priority from the following provisional patent applications: U.S. Application No. 62/061,293, entitled, "Reliable and Efficient Motionless Image Transfer for SIP RTP Media Conferencing" filed October 8, 2014.

BACKGROUND

[0003] Field

[0004] The present system integrates into a Real-time Transport Protocol (RTP) Media

Conferencing System to provide methods of reliable and efficient transport for motionless video images and user device inputs between distributed networked conference participants.

[0005] Description of the Related Art

[0006] This system references the following related art:

[0007] 1. RFC 3261 - SIP: Session Initiation Protocol

[0008] 2. RFC 3550 - RTP: A Transport Protocol for Real-Time Applications

[0009] 3. RFC 3711 - SRTP: The Secure Real Time Transport Protocol

[0010] 4. RFC 5109 - RTP Payload Format for Generic Forward Error Correction

[0011] 5. RFC 5285 - A General Mechanism for RTP Header Extensions

[0012] 6. RFC 2435 - RTP Payload Format for JPEG-compressed Video

[0013] 7. RFC 6466 - IANA Registration for the 'image' Media Type for the Session

Description Protocol (SDP)

[0014] 8. Systems and method for enhancing teleconferencing collaboration, US 7916165 B2

[0015] 9. RTP payload format, US 7876896 B2

SUMMARY

[0016] The present system when integrated into a distributed SIP/RTP Conferencing System provides an efficient, high quality, and reliable transport for motionless video images as well as significantly improve the image quality of single image data frames. The system is well suited for high quality still video images captured from either a digital camera device or a bit map image of all or a portion of a video graphic display memory. The system provides several key improvements to standard RTP video media streaming techniques:

[0017] 1. Transfer of motionless video images using reliable network transport protocols.

[0018] 2. Efficient utilization of network bandwidth and computer processing for large conferences with most participants in listening mode employing motionless video image capture.

[0019] 3. Adds privacy by eliminating unintentional video capture updates of conference participants in listening mode.

[0020] 4. Eliminates distracting motions such as fidgeting with unnecessary video updates from remote conference participants in listening mode.

[0021] 5. Lower bandwidth transfer of high quality motionless video images using standard RTP unreliable network transport protocol with optional error correction.

[0022] 6. Targeted resend of motionless video images using unreliable network transport protocols in routed RTP conferencing systems.

[0023] 7. RTP Extension Header Signaling methods for image file status indications as well as transport of user device inputs such as keyboard and mouse inputs. [0024] 8. Can be integrated into existing routing decision and floor control algorithms to drastically lower network bandwidth consumption and ensure image updates are provided to participant RTP Endpoints entering an existing conference.

[0025] 9. Provides packet image caching in conference routing server to provide image updates to participants when joining an established conference and also to resend images to targeted participants with network transmission failures.

[0026] 10. Supports all motionless image media formats.

[0027] Three methods of motionless image transport are provided:

[0028] 1. RTP with SRTP EKT and optional Error Correction.

[0029] 2. SIP XML with TLS

[0030] 3. Secure FTP

[0031] RTP is used for efficient, low delay, routed conferencing with RTP Header Extension used to provide signaling information and user device input. This method works with point-to-point and meshed conferencing as well as server based conferencing. Reliability of this transport method is improved using optional error correction for the payload and also the use of signaling in the RTP Header Extension to request resending lost packets.

[0032] SIP XML with TLS is primarily used for point-to-point or meshed conferencing but can be practical for smaller server based conferencing as well. This technique uses TLS for reliable delivery of image data but may require a SIP network server to relay the media as well.

[0033] Secure FTP uses a conferencing FTP server to upload motionless images and the RTP Header Extension for image status signaling. Remote conference participants receive RTP Header Extension notifications with image file information for FTP downloading.

[0034] In all motionless image transport methods, associated user inputs such as mouse clicks and coordinates, keyboard inputs, as well as cursor location can be provided with the associated RTP Header Extension. This would be needed for instance with the interaction between remote participants in a desktop or application sharing session.

BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 shows an embodiment of SRTP transport of a motionless image in a Point-to-Point (P2P) RTP media conference without server support.

[0036] Figure 2 shows an embodiment of SIP TLS transport of a motionless image in a

Point-to-Point (P2P) RTP media conference with SIP Proxy/Relay server support.

[0037] Figure 3 shows an embodiment of Secure FTP transport of a motionless image in a Point-to-Point (P2P) RTP media conference with Secure FTP server support.

[0038] Figure 4 shows an embodiment of a SIP/RTP Routing Conference System supporting RTP Motionless Image Transport with server based Caching for image updates and resending and also support for both SIP TLS and Secure FTP Motionless Image

Transport.

[0039] Figure 5 shows the SRTP packet format with RTP Header Extension and Encrypted Key Transport (EKT).

DETAILED DESCRIPTION

[0040] When required, networked media conferencing systems use a variety of techniques for transporting image data to conference participants. Many, possibly the majority, and these techniques are based on encoding image sequences that include algorithms for detecting and encoding inter-frame changes resulting from motion. For motionless images such as a single frame capture from a camera or a bit map graphic image from a computer display memory, video encoding formats, such as H.264 and VP8, are less efficient and result in higher network bandwidths and lower image quality than using motionless image encoding formats such as JPEG. When using motion video encoding formats on motionless images the result will be more network bandwidth, longer delays in image rendering and lower motionless image quality. Additionally, most motion video transport techniques rely solely on unreliable delivery protocols such as RTP which further reduces image quality or even complete loss of ability to render a usable image. Application areas such as high resolution image sharing as is needed for MRI images, X-Ray images, and so on can be seriously compromised and even become unusable when motion video encoding techniques are used in conferencing systems.

[0041] This system addresses situations when shared image information is motionless and motion video encoding formats are not required. Proprietary techniques have been developed to augment conferencing systems to address the need for transport of non- changing image data in conferencing systems. Some systems offer restricted 'presentation' modes that transfer presentation data to the conference server for distribution to the participants and then synchronize conference participants when collaborating on the 'presentation' . These techniques require that the entire presentation be transported across the network from the originator to the server at the very least and can also be forwarded to each conference participant as well. For large presentations, the data transfer can be substantial and also inefficient if the entire presentation is not needed for the conference such as during the collaboration of the presentation development by various contributors on different parts of the overall presentation. Additionally, this technique is not generic and will not work across all platforms and with all applications.

[0042] For generic conference collaboration, a common technique is to share the display memory associated with the entire desktop or application window using a motion video encoding format. In addition to providing no option for reliable network communications, this approach requires continuous network bandwidth and results in lower quality image rendering of motionless video. An illustrative example that contrasts the use of standard motion video encoding of motionless video is with a medium-high resolution 800 by 600 pixel image with 24-bit pixel encoding. The raw image frame size is over 11 megabits. Processing this image using the current global motion video standard H.264 can be expected to require a minimum of a several megabit spike to transport the reference image frame and several 100 kilobit per seconds of network bandwidth for the associated non-reference frames. Additionally more several megabit spikes on the network will occur since the reference frame must be regenerated periodically because of accumulated encoding errors as well as every time a new participant joins the conference and requires a reference frame to render an image from the encoded H.264 data. By contrast, this system only requires a single transmission of the encoded motionless video image. When used with a conferencing server equipped with the system's image caching mechanism, new conference participants receive targeted encoded images from the server instead of requiring the source participant to send the encoded image and then having the image forwarded to all conference participants. In this example the system would require significantly less than the H.264 reference frame for a better quality motionless encoding and only a single transmission on the data network resulting in a total network bandwidth requirement of far less than 1 megabits per conference participant. For a 60 minute conference this could be expected to reduce the required network bandwidth to far less than 1% of the network bandwidth required for the current motion encoding technique with H.264.

[0043] The present system when integrated into a distributed SIP/RTP Conferencing System provides an efficient and reliable transport for motionless images as well as significantly improving the image quality of single image data frames. The system is well suited for high quality still video images captured from either a digital camera device or a bit map image of all or a portion of a video graphic display memory. The system provides several key improvements to standard RTP video media streaming techniques:

[0044] 1. Efficient utilization of network bandwidth and computer processing for large conferences with most participants in listening mode employing motionless video image capture.

[0045] 2. Adds privacy by eliminating unintentional video capture updates of conference participants in listening mode.

[0046] 3. Eliminates distracting motions such as fidgeting with unnecessary video updates from remote conference participants in listening mode.

[0047] 4. Transfer of motionless video images using reliable network transport protocols based on TCP such as SIP XML messaging or FTP using secure TLS connections. [0048] 5. Lower bandwidth transfer of high quality motionless video images using unreliable network transport protocols such as RTP secured with SRTP and enhanced with optional error correction for recovery from packet losses.

[0049] 6. Targeted resend of motionless video images using unreliable network transport protocols in routed RTP conferencing systems for more efficient recovery from packet losses.

[0050] 7. RTP Extension Header Signaling methods for image file status indications as well as transport of user device inputs such as keyboard and mouse inputs.

[0051] 8. Can be integrated into existing routing decision and floor control algorithms to drastically lower network bandwidth consumption and ensure image updates are provided to participant RTP Endpoints entering an existing conference.

[0052] 9. Provides packet image caching in conference routing server to provide image updates to participants when joining an established conference and also to resend images to targeted participants with network transmission failures.

[0053] 10. Supports all motionless image media formats.

[0054] Three methods of motionless image transport are provided by the system:

[0055] 1. RTP with SRTP EKT and optional Error Correction.

[0056] 2. SIP XML with TLS

[0057] 3. Secure FTP

[0058] RTP is used for efficient, low delay, routed conferencing with RTP Header Extension used to provide signaling information and user device input. This method works with both point-to-point (P2P) and server based conferencing. Reliability of this transport works with both point-to-p is improved using optional error correction for the transport method is improved for reliable transport using optical error correction for the payload and also the use of signaling in the RTP Header Extension to request resending lost packets. Reliable transport is further improved with implementation of a sliding window protocol either in the RTP routing server or as part of the image data transport layer.

[0059] Figure 1 depicts a P2P conferencing system using only RTP for the motionless image transport. Conference Participant SP sources an image encoded in a motionless format such as JPEG by packetizing the formatted image into RTP packets using SRTP to secure the media transport and SRTP Encrypted Key Transport (EKT) to support routing of SRTP packets at the RTP network level.

[0060] SIP XML with TLS is primarily used for point-to-point or meshed conferencing but can be practical for smaller server based conferencing as well. This technique uses TLS for reliable delivery of image data but may require a SIP network server to relay the media as well.

[0061] Figure 2 depicts a P2P conferencing system using SIP TLS for reliable image transport.

[0062] Secure FTP uses a conferencing FTP server to upload motionless images and the RTP Header Extension for image status signaling. Remote conference participants receive RTP Header Extension notifications with image file information for FTP downloading.

[0063] Figure 3 depicts a P2P conferencing system using FTP for reliable image transport. The FTP server connection information can be obtained by the conference participants either with SIP SDP or a conference administration server.

[0064] In all motionless image transport methods, associated user inputs such as mouse clicks and coordinates, keyboard inputs, as well as cursor location can be provided with the associated RTP Header Extension. This would be needed for instance with the interaction between remote participants in a desktop or application sharing session.

[0065] For larger conferences or conferences requiring deterministic and reliable network performance, a routing conference server is used as depicted in Figure 4. A routed conferencing system will likely use the SRTP with EKT image transport method for efficiency and low communication delays. For network elements with intermittent packet loss issues, error correction may be also used. However for network elements with significant and consistent packet loss issues, reliable packet delivery can be provided with either the Secure FTP transport method or the SIP TLS method. For conferences with a large number of participants, the network elements with significant packet loss issues would likely use the FTP transport method for the most reliable method of transport and the lowest implementation complexity and also lower over communication delays.

[0066] The system also provides packet image caching in conference routing servers to provide efficient image updates to participants when joining an established conference and also to resend images to targeted participants with network transmission failures. In a motion encoding conferencing system, the encoded reference frame must be generated and resent to each conference participant each time a new participant joins a conference. With both the motionless encoding and image caching of the system, only the new participant receives the encoded image. A conference with 20 attendees illustrates this. With motion encoding a good resolution image would expect on the order of 50 times more network bandwidth on a conferencing server than with the use of this system. As the number of conference attendees grows so does the network bandwidth burden on the conference server. However it remains constant at the same minimized level when the system is used.

Claims

CLAIMS It is claimed:

1. An efficient and reliable transport for motionless images in a standards based RTP (Real-time Transport Protocol) Media Conferencing System, the method comprising of the acts of:

Secure transport of motionless images over an IP network using TCP (Transport Control Protocol) or enhanced RTP.

2. The method of claim 1, wherein the method of transport is Secure FTP using an FTP file repository common to all participants in a conference.

3. The method of claim 1, wherein the method of transport is SIP TLS using XML messaging.

4. The method of claim 1, wherein the method of transport is RTP.

5. The method of claim 4 using optional error correction coding to increase reliability in packet transport.

6. The method of claims 2, 3, and 4 where the RTP Header Extension is used to provide user device inputs, such as cursor location and mouse events and keyboard events, from the conference participant originating the image to conference participants receiving the image when the image is used in conference collaboration.

7. The method of claim 4, wherein the data associated with each image is cached by a sending conference participant or conference routing server to be forwarded to conference participants joining an established conference.

8. The method of claim 4, wherein the data associated with each image is cached by a sending conference participant or conference routing server to be forwarded to conference participants requesting the image using RTP Header Extension signaling.

9. The method of claim 1, where SIP TLS is used with XML messaging to send user device inputs, such as mouse events and keyboard events, from a conference participant receiving the image to the conference participant originating the image for conference collaboration.

10. The method of claim 1, where RTP Header Extension signaling is used to send user device inputs, such as mouse events and keyboard events, from a conference participant receiving the image to the conference participant originating the image for conference collaboration.

11. The method of claim 10 where the RTP Header Extension signaling is labeled to allow for identical messages to be sent and filtered by the recipient to ensure that singular events are only processed once

12. The method of claim 1 where the image attributes are exchanged using SDP (Session Description Protocol) during SIP dialogs such as the SIP INVITE.

13. The method of claims 1 and 12 wherein the RTP attributes associated with the

m=image media type are provided in the SIP SDP.